Home

Awesome

OpenOOD-VLM: Benchmarking Generalized OOD Detection with Vision-Language Models

This repository release the official codes of our works for out-of-distribution detection with Vision-Language Models. For reference, we also reproduce some representative methods of OOD detection.

šŸ”„ News

āš™ļø Dependencies and Installation

<details> <summary>Follow OpenOOD to set up the environment, or use our provided instructions below.</summary>

pip install git+https://github.com/YBZH/OpenOOD-VLM

</details>

šŸ“· Datasets

We also follow OpenOOD to manage the training and testing datasets. If you only use our evaluator, the benchmarks for evaluation will be automatically downloaded by the evaluator (again check out this tutorial). If you would like to also use OpenOOD-VLM for training, you can get all data with our downloading script. Note that ImageNet-1K training images should be downloaded from its official website.

Besides datasets used in OpenOOD, we also provide evaluation on some popular OOD datasets iNaturalist, SUN, Places, and Texture curated by Huang et al. 2021. Please follow instruction from the this repository to download the subsampled datasets where semantically overlapped classes with ImageNet-1k are removed.

Our codebase accesses the datasets from ./data/ and pretrained models from ./results/checkpoints/ by default.

ā”œā”€ā”€ ...
ā”œā”€ā”€ data
ā”‚   ā”œā”€ā”€ benchmark_imglist
ā”‚   ā”œā”€ā”€ images_classic
ā”‚   ā””ā”€ā”€ images_largescale
ā”œā”€ā”€ openood
ā”œā”€ā”€ results
ā”‚   ā”œā”€ā”€ checkpoints
ā”‚   ā””ā”€ā”€ ...
ā”œā”€ā”€ scripts
ā”œā”€ā”€ main.py
ā”œā”€ā”€ ...
<details> <summary><b>Supported Datasets for Out-of-Distribution Detection</b></summary>
</details>

šŸš€ Training/Testing

We provide training and evaluation scripts for all the methods we support in scripts folder.

sh scripts/ood/lapt/test.sh
sh scripts/ood/mcm/official.sh
sh scripts/ood/neglabel/official.sh
<!-- We release the **official codes** of our works for OOD detection with Vision-Language Models, including: > - [x] [ECCV2024: LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models](https://arxiv.org/pdf/2407.08966) with [Bash](scripts/ood/lapt/test.sh) and [Checkpoint](https://drive.google.com/file/d/1uHvqAraQDBuc_IjbqyLgSdjFDxesTAA7/view?usp=sharing) </details> We will also reproduce some interesting works for OOD detection with Vision-Language Models, including: > - [x] [NIPS2022: Delving into out-of-distribution detection with vision-language representations](https://arxiv.org/pdf/2211.13445) with [Bash](scripts/ood/mcm/official.sh) > - [x] [ICLR2024: Negative Label Guided OOD Detection with Pretrained Vision-Language Models](https://arxiv.org/pdf/2403.20078) with [Bash](scripts/ood/neglabel/official.sh) </details> -->

šŸ“– Citation

If you find our repository useful for your research, please consider citing our paper:


@article{zhang2024lapt,
  title={LAPT: Label-driven Automated Prompt Tuning for OOD Detection with Vision-Language Models},
  author={Zhang, Yabin and Zhu, Wenjie and He, Chenhang and Zhang, Lei},
  journal={Proceedings of the european conference on computer vision (ECCV)},
  year={2024}
}


šŸ™ Acknowledgement