Awesome

SCOOD-UDG (ICCV 2021)

This repository is the official implementation of the paper:

Semantically Coherent Out-of-Distribution Detection<br> Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, Ziwei Liu<br> Proceedings of the IEEE International Conference on Computer Vision (ICCV 2021)<br>

Dependencies

We use conda to manage our dependencies, and CUDA 10.1 to run our experiments.

You can specify the appropriate cudatoolkit version to install on your machine in the environment.yml file, and then run the following to create the conda environment:

conda env create -f environment.yml
conda activate scood

SC-OOD Dataset

The SC-OOD dataset introduced in the paper can be downloaded here.

Our codebase accesses the dataset from the root directory in a folder named data/ by default, i.e.

├── ...
├── data
│   ├── images
│   └── imglist
├── scood
├── test.py
├── train.py
├── ...

Training

The entry point for training is the train.py script. The hyperparameters for each experiment is specified by a .yml configuration file (examples given in configs/train/).

All experiment artifacts are saved in the specified args.output_dir directory.

python train.py \
    --config configs/train/cifar10_udg.yml \
    --data_dir data \
    --output_dir output/cifar10_udg

Testing

Evaluation for a trained model is performed by the test.py script, with its hyperparameters also specified by a .yml configuration file (examples given in configs/test/)

Within the configuration file, you can also specify which post-processing OOD method to use (e.g. ODIN or Energy-based OOD detector (EBO)).

The evaluation results are saved in a .csv file as specified.

python test.py \
    --config configs/test/cifar10.yml \
    --checkpoint output/cifar10_udg/best.ckpt \
    --data_dir data \
    --csv_path output/cifar10_udg/results.csv

Results

We report the mean ± std results from the current codebase as follows, which match the performance reported in our original paper.

CIFAR-10 (+ Tiny-ImageNet) Results

You can run the following script (specifying the data and output directories) which perform training + testing for our main experimental results:

CIFAR-10, UDG

bash scripts/cifar10_udg.sh data_dir output_dir

CIFAR-10 (+ Tiny-ImageNet), ResNet18

CIFAR-10 (+ Tiny-ImageNet), DenseNet

CIFAR-10 (+ Tiny-ImageNet), WideResNet

CIFAR-100 (+ Tiny-ImageNet) Results

You can run the following script (specifying the data and output directories) which perform training + testing for our main experimental results:

CIFAR-100, UDG

bash scripts/cifar100_udg.sh data_dir output_dir

CIFAR-100 (+ Tiny-ImageNet), ResNet18

CIFAR-100 (+ Tiny-ImageNet), DenseNet

CIFAR-100 (+ Tiny-ImageNet), WideResNet

Note: The work was originally built on the company's own deep learning framework, based on which we report all the results in the paper. We extracted all related code and built this standalone version for release, and checked that most of the results can be reproduced. We noticed that CIFAR-10 can easily match the paper results, but CIFAR-100 benchmark might have a few differences, perhaps due to some minor difference in framework modules and some randomness. We are currently enhancing our codebase and exploring udg on large-scale datasets.

License and Acknowledgements

This project is open-sourced under the MIT license.

The codebase is refactored by Ang Yi Zhe, and maintained by Jingkang Yang and Ang Yi Zhe.

Citation

If you find our repository useful for your research, please consider citing our paper:

@InProceedings{yang2021scood,
    author = {Yang, Jingkang and Wang, Haoqi and Feng, Litong and Yan, Xiaopeng and Zheng, Huabin and Zhang, Wayne and Liu, Ziwei},
    title = {Semantically Coherent Out-of-Distribution Detection},
    booktitle = {Proceedings of the IEEE International Conference on Computer Vision},
    year = {2021}
}