Awesome

ReweightOOD: Loss Reweighting for Distance-based OOD Detection

This codebase provides a Pytorch implementation of:

ReweightOOD: Loss Reweighting for Distance-based OOD Detection.

Sudarshan Regmi, Bibek Panthi, Yifei Ming, Prashnna K. Gyawali, Danail Stoyanov, Binod Bhattarai

Abstract

Out-of-Distribution (OOD) detection is crucial for ensuring safety and reliability of neural networks in critical applications. Distance-based OOD detection is based on the assumption that OOD samples are mapped far from In-Distribution (ID) clusters in embedding space. A recent approach for obtaining OOD-detection-friendly embedding space has been contrastive optimization of pulling similar pairs and pushing apart dissimilar pairs. It assigns equal significance to all similarity instances with the implicit objective of maximizing the mean proximity between samples with their corresponding hypothetical class centroids. However, the emphasis should be directed towards reducing the Minimum Enclosing Sphere (MES) for each class and achieving higher inter-class dispersion to effectively mitigate the potential for ID-OOD overlap. Optimizing low-signal dissimilar pairs might potentially act against achieving maximal inter-class dispersion while less-optimized similar pairs prevent achieving smaller MES. Based on this, we propose a reweighting scheme ReweightOOD, that adopts the similarity optimization which prioritizes the optimization of less-optimized contrasting pairs while assigning lower importance to already well-optimized contrasting pairs. Such a reweighting scheme serves to minimize the MES for each class while achieving maximal inter-class dispersion. Experimental results on a challenging CIFAR100 benchmark using ResNet-18 network demonstrate that ReweightOOD outperforms supervised contrastive loss by a whopping 38% in the average FPR metric. In various classification datasets, our method provides a promising solution for enhancing OOD detection capabilities in neural networks.

ReweightOOD highlights in OpenOODv1.5 leaderboard [Updated: June 2024]

➤ ranks 2nd in the far-OOD AUROC metric on CIFAR10 datasets.
➤ ranks 1st in the far-OOD AUROC metric on CIFAR100 datasets.
➤ ranks 5th in the far-OOD AUROC metric on ImageNet200 datasets (OOD).
➤ ranks 2nd in the far-OOD AUROC metric on ImageNet200 datasets (FSOOD).
➤ ranks 9th in the far-OOD AUROC metric on ImageNet-1k datasets (OOD).
➤ ranks 6th in the far-OOD AUROC metric on ImageNet-1k datasets (FSOOD).

Follow OpenOOD official instruction to complete the setup.

pip install git+https://github.com/Jingkang50/OpenOOD

Method Illustration

Example Scripts for Training and Inference

Use the following scripts for training and inferencing the ReweightOOD model on different datasets:

• CIFAR-10:

  bash scripts/ood/reweightood/cifar10_train_reweightood.sh
  bash scripts/ood/reweightood/cifar10_test_reweightood.sh
  

• CIFAR-100:

  bash scripts/ood/reweightood/cifar100_train_reweightood.sh
  bash scripts/ood/reweightood/cifar100_test_reweightood.sh
  

• ImageNet-200:

  bash scripts/ood/reweightood/imagenet200_train_reweightood.sh
  bash scripts/ood/reweightood/imagenet200_test_reweightood.sh
  

• ImageNet-1k:

  bash scripts/ood/reweightood/imagenet_train_reweightood.sh
  bash scripts/ood/reweightood/imagenet_test_reweightood.sh
  

Results

• CIFAR-10 (OOD):

• CIFAR-100 (OOD):

• ImageNet-200 (OOD):

• ImageNet-200 (FSOOD):

• ImageNet-1k (OOD):

• ImageNet-1k (FSOOD):

Pre-trained checkpoints

Pre-trained models are available in the given links:

• CIFAR-10 [Google Drive]: ResNet-18 backbone trained on CIFAR-10 datasets with ReweightOOD objective across 3 trials.
• CIFAR-100 [Google Drive]: ResNet-18 backbone trained on CIFAR-100 datasets with ReweightOOD objective across 3 trials.
• ImageNet-200 [Google Drive]: ResNet-18 backbone trained on ImageNet-200 datasets with ReweightOOD objective across 3 trials.
• ImageNet-1k [Google Drive]: ResNet-50 backbone fine-tuned on ImageNet-1k datasets with ReweightOOD objective across 1 trial.

Please consider citing our work if you find it useful.

@InProceedings{Regmi_2024_CVPR_1,
    author    = {Regmi, Sudarshan and Panthi, Bibek and Ming, Yifei and Gyawali, Prashnna K and Stoyanov, Danail and Bhattarai, Binod},
    title     = {ReweightOOD: Loss Reweighting for Distance-based OOD Detection},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2024},
    pages     = {131-141}
}

Acknowledgment

This codebase builds upon OpenOOD.