Home

Awesome

A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning

Michael Kirchhof^, Karsten Roth^, Zeynep Akata, Enkelejda Kasneci (^) equal contribution

alt text

Proxy-based Deep Metric Learning (DML) learns deep representations by embedding images close to their class representatives (proxies), commonly with respect to the angle between them. However, this disregards the embedding norm, which can carry additional beneficial context such as class- or image-intrinsic uncertainty. In addition, proxy-based DML struggles to learn class-internal structures. To address both issues at once, we introduce non-isotropic probabilistic proxy-based DML. We model images as directional von Mises-Fisher (vMF) distributions on the hypersphere that can reflect image-intrinsic uncertainties. Further, we derive non-isotropic von Mises-Fisher (nivMF) distributions for class proxies to better represent complex class-specific variances. To measure the proxy-to-image distance between these models, we develop and investigate multiple distribution-to-point and distributionto-distribution metrics. Each framework choice is motivated by a set of ablational studies, which showcase beneficial properties of our probabilistic approach to proxy-based DML, such as uncertainty-awareness, better behaved gradients during training, and overall improved generalization performance. The latter is especially reflected in the competitive performance on the standard DML benchmarks, where our approach compares favourably, suggesting that existing proxy-based DML can significantly benefit from a more probabilistic treatment.

Accepted to ECCV 2022.

Link: https://arxiv.org/pdf/2207.03784.pdf

BibTeX:

@inproceedings{Kirchhof2022ANP,
  title={A Non-isotropic Probabilistic Take on Proxy-based Deep Metric Learning},
  author={Michael Kirchhof and Karsten Roth and Zeynep Akata and Enkelejda Kasneci},
  booktitle={European Conference on Computer Vision},
  year={2022}
}

Requirements.

This repository was tested around

To install, with a package manager of your choice, simply install following e.g.

conda create -n sample_env python=3.9
conda activate sample_env
conda install matplotlib joblib scikit-image scikit-learn scipy pandas pillow termcolor
conda install pytorch torchvision faiss-gpu cudatoolkit=11.3 -c pytorch
pip install wandb pretrainedmodels timm

Our results are computed using standard Deep Metric Learning benchmarks, in particular CUB200-2011, CARS196 and Stanford Online Products. For help on how to install these and correctly set them up, we refer to Link. To be directly loadable, we expect each dataset to have the following structure:

For CUB200-2011/CARS196:

cub200/cars196
└───images
|    └───001.Black_footed_Albatross
|           │   Black_Footed_Albatross_0001_796111
|           │   ...
|    ...

For Stanford Online Products:

online_products
└───images
|    └───bicycle_final
|           │   111085122871_0.jpg
|    ...
|
└───Info_Files
|    │   bicycle.txt
|    │   .

Training a probabilistic DML model.

In it's structure, this repository follows the layout and code introduced in "Revisiting Training Strategies and Generalization Performance in Deep Metric Learning", Roth et al. (2020).

Exemplary training runs are provided in example_runs.sh. In general, training and loss-selection is done following Roth et al. 2020. As an example, to train a probabilistic and non-isotropic EL-nivMF regularized ProxyAnchor loss on the CARS196 dataset, simply call

parent_folder="path_to_parent_folder"
project="W&B project name"
gpu="gpu_id"

python main.py --loss proxyvmf_panc --seed 0 --gpu $gpu --source_path $parent_folder --project $project --log_online --group Cars196_Prob_RN50_dim128 --dataset cars196 \
--n_epochs 300 --tau 60 80 --gamma 0.1 --no_train_metrics --embed_dim 128 \
--loss_proxyvmf_proxylrmulti 7500 --loss_proxyvmf_conclrmulti 7500 --loss_proxyvmf_concentration 9 --loss_proxyvmf_temp 0.003 --loss_proxyvmf_templrmulti 200 --loss_proxyvmf_warmstart --loss_proxyvmf_n_samples 5 --loss_proxyvmf_guidance_w 0.3

which fixed the learning rate multipliers for each proxy and vMF kappas/concentrations (--loss_proxyvmf_proxy/conclrmulti), a starting length for each proxy (--loss_proxyvmf_concentration), temperature regularizer for the log-PPK (--loss_proxyvmf_temp and associated lr multiplier --loss_proxyvmf_templrmulti), the number of samples to draw from our vMFs (--loss_proxyvmf_n_samples) and finally the weight on the additional ProxyAnchor loss on top of the EL-nivMF objective (--loss_proxyvmf_guidance_w). Finally, --loss_proxyvmf_warmstart simply denotes one additional epoch at the beginning used to only update the proxies/kappas before training the full pipeline.

In general, all relevant parameters for the EL-nivMF objective are made available in parameters.py > loss_specific_parameters() > loss_proxyvmf_..., with the explicit implementation provided in criteria/proxyvmf_panc.py, and with utilities.misc.log_ppk_vmf_vec(...) containing the vectorized log-PPK implementation, and utilities.vmf_sampler the associated vMF sampler implementation.

Finally, borrowed from Roth et al. 2020, a quick explanation of some other relevant flags (see also parameters.py for a list of all possible parameters and flags and a quick description):

Some Notes: