Awesome

TAG: Text Prompt Augmentation for Zero-Shot Out-of-Distribution Detection

This is the offical repository for our TAG and the presentation can be found here.

Illustration

Data Preparation

We consider the following (in-distribution) datasets: CIFAR-100, ImageNet-100, and ImageNet-1k.

Small-scale OOD datasets For small-scale ID (e.g. CIFAR-100), we use SVHN, Textures (dtd), Places365, LSUN and iSUN.

OOD datasets can be downloaded via the following links (source: ATOM and CIDER):

• SVHN: download it and place it in the folder of datasets/small_OOD_dataset/svhn. Then run python utils/select_svhn_data.py to generate test subset.
• Textures: download it and place it in the folder of datasets/small_OOD_dataset/dtd.
• Places365: download it and place it in the folder of datasets/ood_datasets/places365/test_subset. We randomly sample 10,000 images from the original test dataset.
• LSUN-C: download it and place it in the folder of datasets/small_OOD_dataset/LSUN.
• iSUN: download it and place it in the folder of datasets/small_OOD_dataset/iSUN.

Large-scale OOD datasets For large-scale ID (e.g. ImageNet-100), we use the curated 4 OOD datasets from OpenImage-O, Texture, iNaturalist, and ImageNet-O. (source: ViM)

• ImageNet. The ILSVRC 2012 dataset as In-distribution (ID) dataset.
• ImageNet-100.
• OpenImage-O. The OpenImage-O dataset is a subset of the OpenImage-V3 testing set. The filelist is here.
• Texture. We rule out four classes that coincides with ImageNet. The filelist used in the paper is here.
• iNaturalist. Follow the instructions in the link to prepare the iNaturalist OOD dataset.
• ImageNet-O. Follow the guide to download the ImageNet-O OOD dataset.

Evaluation

1. extract features

   python eval/extract_features.py  # specify model name ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101']  
   

2. evaluation

   python eval/eval_ood_cifar_100.py --model_name  ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101'] --shuffle --M [int_number_of_augmentations] --temp [temperature] --score_name ['MSP', 'Max-Logit','Energy','GEN']
   python eval/eval_ood_imagenet_100.py --model_name  ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101'] --shuffle --M [int_number_of_augmentations] --temp [temperature] --score_name ['MSP', 'Max-Logit','Energy','GEN'] 
   python eval/eval_ood_imagenet_1k.py --model_name  ['ViT-B/16', 'ViT-B/32','ViT-L/14', 'RN50', 'RN101'] --shuffle --M [int_number_of_augmentations] --temp [temperature] --score_name ['MSP', 'Max-Logit','Energy','GEN']  
   

3. texture embedding analysis

    python eval/SVD_text_embeddings.py
   

Citation

@inproceedings{Liu2024TAG,
title = {TAG: Text Prompt Augmentation for Zero-Shot Out-of-Distribution Detection},
author = {Liu, Xixi and Christopher, Zach},
booktitle = {European Conference on Computer Vision},
year = {2024}
}

Acknowledgement

Our data preparartion code is adapted from ViM and CIDER repos, thanks a lot for their great work!