Awesome

Ensemble of Self Supervised Leave-out Classifiers

This is an unofficial pytorch implementation of Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers.

Requirements

• Python 3.7+
• PyTorch 1.10.2
• torchvision 0.11.3
• progress
• matplotlib
• numpy

Preparation

Download five out-of-distributin datasets provided by ODIN:

• Tiny-ImageNet (crop)
• Tiny-ImageNet (resize)
• LSUN (crop)
• LSUN (resize)
• iSUN

Here is an example code of downloading Tiny-ImageNet (crop) dataset. In the root directory, run

mkdir data
cd data
wget https://www.dropbox.com/s/avgm2u562itwpkl/Imagenet.tar.gz
tar -xvzf Imagenet.tar.gz
cd ..

Usage

Pre-trained Models

We provide download links of four types of pre-trained models.

• DenseNet-BC trained on CIFAR-10
• DenseNet-BC trained on CIFAR-100
• Wide ResNet trained on CIFAR-10
• Wide ResNet trained on CIFAR-100

Each type of models contains 5 models for 5-fold. Here is an example code of downloading DenseNet-BC trained on CIFAR-10. In the root directory, run

mkdir checkpoints
cd checkpoints
wget https://www.dropbox.com/s/mo2xay9rpfk8emu/cifar10_dense.tar.gz
tar -xvzf cifar10_dense.tar.gz
cd ..

Train single model (Optional)

Train DenseNet on the first fold of CIFAR-10.

python train.py -c checkpoints/cifar10_fold_1_dense_checkpoint --fold 1

Trained model will be saved at checkpoints/cifar10_fold_1_dense_checkpoint.

Train all models (Optional)

python train_all.py

This script will train models of DenseNet/WideResNet on 5-fold CIFAR-10/100 which results in 20 models. Trained model will be saved at checkpoints.

Test

Use 5-fold ensemble and ODIN to detect OOD samples. For example, to test DenseNet-BC trained on CIFAR-10 where TinyImageNet (crop) is the out-of-distribution dataset, please run

python test.py --in-dataset cifar10 --out-dataset Imagenet --magnitud 0.002

The temperature is set as 1000, and perturbation magnitude is set as 0.002.

Note: Please choose arguments according to the following.

args

• args.in_dataset: the arguments of in-distribution datasets are shown as follows

  In-Distribution Datasets	args.in_dataset
  CIFAR-10	cifar10
  CIFAR-100	cifar100

• args.out_dataset: the arguments of out-of-distribution datasets are shown as follows

  Out-of-Distribution Datasets	args.out_dataset
  Tiny-ImageNet (crop)	Imagenet
  Tiny-ImageNet (resize)	Imagenet_resize
  LSUN (crop)	LSUN
  LSUN (resize)	LSUN_resize
  iSUN	iSUN

• args.magnitude: the noise magnitude can be found below. You can use these values to reproduce the retults. Please notice that these values are not optimal .

  Out-of-Distribution Datasets	DenseNet on CIFAR-10	DenseNet on CIFAR-100	WideResNet on CIFAR-10	WideResNet on CIFAR-100
  Tiny-ImageNet (crop)	0.002	0.002	0.003	0.002
  Tiny-ImageNet (resize)	0.002	0.002	0.003	0.002
  LSUN (crop)	0.002	0.003	0.001	0.002
  LSUN (resize)	0.002	0.003	0.002	0.002

• args.wide: the arguments of network choices are shown as follows

  Nerual Network Models	args.wide
  DenseNet	False
  WideResNet	True

• args.temperature: temperature is set to 1000 in all cases.

Results

All values are percentages. ↑ indicates larger value is better, and ↓ indicates lower value is better. Each value cell is in "Paper/Our Implementation" format.

DenseNet on CIFAR-10

DenseNet on CIFAR-100

WideResNet on CIFAR-10

WideResNet on CIFAR-100

References

• [1]: A. Vyas, N. Jammalamadaka and et al. "Out-of-Distribution Detection Using an Ensemble of Self Supervised Leave-out Classifiers", in ECCV, 2018.
• [2]: S. Liang, Y. Li and R. Srikant. "Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks", in ICLR, 2018.