Home

Awesome

šŸŒŽ [CVPR 2023] BalPoE-CalibratedLT

by Emanuel Sanchez Aimar, Arvi Jonnarth, Michael Felsberg, Marco Kuhlmann

This repository contains the official Pytorch implementation of Balanced Product of Calibrated Experts for Long-Tailed Recognition at CVPR 2023.

šŸŽ¬Video | šŸ’»Slides | šŸ”„Poster | ArXiv preprint

Method

<p align="center"> <img src='assets/balpoe_calibrated_framework.png' align="center"> </p>

Abstract

Many real-world recognition problems are characterized by long-tailed label distributions. These distributions make representation learning highly challenging due to limited generalization over the tail classes. If the test distribution differs from the training distribution, e.g. uniform versus long-tailed, the problem of the distribution shift needs to be addressed. A recent line of work proposes learning multiple diverse experts to tackle this issue. Ensemble diversity is encouraged by various techniques, e.g. by specializing different experts in the head and the tail classes. In this work, we take an analytical approach and extend the notion of logit adjustment to ensembles to form a Balanced Product of Experts (BalPoE). BalPoE combines a family of experts with different test-time target distributions, generalizing several previous approaches. We show how to properly define these distributions and combine the experts in order to achieve unbiased predictions, by proving that the ensemble is Fisher-consistent for minimizing the balanced error. Our theoretical analysis shows that our balanced ensemble requires calibrated experts, which we achieve in practice using mixup. We conduct extensive experiments and our method obtains new state-of-the-art results on three long-tailed datasets: CIFAR-100-LT, ImageNet-LT, and iNaturalist-2018.

Getting Started

Prerequisites

BalPoE-CalibratedLT is built on pytorch and a handful of other open-source libraries.

To install the required packages, you can create a conda environment:

conda create --name balpoe python=3.8

then use pip to install required packages:

pip install -r requirements.txt

Hardware requirements

4 GPUs with >= 24G GPU RAM are recommended (for large datasets). Otherwise, the model with more experts may not fit in, especially on datasets with more classes (the FC layers will be large). We do not support CPU training at the moment.

Datasets

Four benchmark datasets

Txt files

data_txt
ā”œā”€ā”€ ImageNet_LT
ā”‚   ā”œā”€ā”€ ImageNet_LT_backward2.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_backward5.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_backward10.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_backward25.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_backward50.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_forward2.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_forward5.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_forward10.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_forward25.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_forward50.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_test.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_train.txt
ā”‚   ā”œā”€ā”€ ImageNet_LT_uniform.txt
ā”‚   ā””ā”€ā”€ ImageNet_LT_val.txt
ā””ā”€ā”€ iNaturalist18
    ā”œā”€ā”€ iNaturalist18_backward2.txt
    ā”œā”€ā”€ iNaturalist18_backward3.txt
    ā”œā”€ā”€ iNaturalist18_forward2.txt
    ā”œā”€ā”€ iNaturalist18_forward3.txt
    ā”œā”€ā”€ iNaturalist18_train.txt
    ā”œā”€ā”€ iNaturalist18_uniform.txt
    ā””ā”€ā”€ iNaturalist18_val.txt 

Usage

CIFAR100-LT

Training

python train.py -c configs/mixup/standard_training/config_cifar100_ir100_bs-experts.json --seed 1
python train.py -c configs/mixup/long_training/config_cifar100_ir100_bs-experts.json --seed 1
python train.py -c configs/mixup/long_training/config_cifar100_ir100_bs-experts.json --tau_list limits --num_experts 7 --seed 1

Evaluate

python test.py -r checkpoint_path

where checkpoint_path should be of the form CHECKPOINT_DIR/checkpoint-epoch[LAST_EPOCH].pth, where LAST_EPOCH is 200 and 400 for standard and long training, respectively.

python test_all_cifar.py -r checkpoint_path [--posthoc_bias_correction]

Optional: use --posthoc_bias_correction to adjust logits with known test prior.

iNaturalist 2018

Training

python train.py -c configs/mixup/standard_training/config_iNaturalist_resnet50_bs-experts.json
python train.py -c configs/mixup/long_training/config_iNaturalist_resnet50_bs-experts.json

Evaluate

python test.py -r checkpoint_path

where checkpoint_path should be of the form CHECKPOINT_DIR/checkpoint-epoch[LAST_EPOCH].pth, where LAST_EPOCH is 100 and 400 for standard and long training, respectively.

python test_all_inat.py -r checkpoint_path [--posthoc_bias_correction]

Optional: use --posthoc_bias_correction to adjust logits with known test prior.

ImageNet-LT

Training

python train.py -c configs/mixup/standard_training/config_imagenet_lt_resnext50_bs-experts.json
python train.py -c configs/mixup/long_training/config_imagenet_lt_resnext50_bs-experts.json

Evaluate

python test.py -r checkpoint_path

where checkpoint_path should be of the form CHECKPOINT_DIR/checkpoint-epoch[LAST_EPOCH].pth, where LAST_EPOCH is 180 and 400 for standard and long training, respectively.

python test_all_imagenet.py -r checkpoint_path

Optional: use run the following command to accomodate for a known test prior.

python test_all_imagenet.py -r checkpoint_path --posthoc_bias_correction

Citation

If you find our work inspiring or use our codebase in your research, please cite our work.

@InProceedings{SanchezAimar2023BalPoE_CalibratedLT,
    author    = {Sanchez Aimar, Emanuel and Jonnarth, Arvi and Felsberg, Michael and Kuhlmann, Marco},
    title     = {Balanced Product of Calibrated Experts for Long-Tailed Recognition},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2023},
    pages     = {19967-19977}
}

Acknowledgements

Our codebase is based on several open-source projects, particularly:

This is a project based on this pytorch template.