Home

Awesome

[ECCV 2024] Uncertainty Calibration with Energy Based Instance-wise Scaling in the Wild Dataset

Official implementation of "Uncertainty Calibration with Energy Based Instance-wise Scaling in the Wild Dataset (ECCV '24)" Mijoo Kim, and Junseok Kwon

arXiv Hits

<!-- **πŸ“’Code will be released soon !** -->

Abstract

<p align='center'> <img src='./figures/pipeline.png' width='900'/> </p>

In this paper, we investigate robust post-hoc uncertainty calibration methods for DNNs within the context of multi-class classification tasks. While previous studies have made notable progress, they still face challenges in achieving robust calibration, particularly in scenarios involving out-of-distribution (OOD). We identify that previous methods lack adaptability to individual input data and struggle to accurately estimate uncertainty when processing inputs drawn from the wild dataset. To address this issue, we introduce a novel instance-wise calibration method based on an energy model. Our method incorporates energy scores instead of softmax confidence scores, allowing for adaptive consideration of DNN uncertainty for each prediction within a logit space.

Run demo

(1) Download datasets

πŸ“¦Energy-Calibration
 β”œβ”€β”€ πŸ“‚data
 β”‚   β”œβ”€β”€πŸ“‚cifar10
 β”‚   β”œβ”€β”€πŸ“‚cifar10C
 β”‚   β”œβ”€β”€πŸ“‚SVHN
 β”‚   β””β”€β”€πŸ“‚dtd
 β”œβ”€β”€ ...

(2) Environment setup

git clone https://github.com/mijoo308/Energy-Calibration.git
cd Energy-Calibration
conda create -n energycal python=3.8
conda activate energycal
pip install -r requirements.txt

(3) Run

python main.py --gpu --ddp

If you already have logit files, you can use the following flags:

(4) Result

After running the main script, you will see the results in your terminal and the results will be saved in the result folder:

πŸ“¦Energy-Calibration
 β”œβ”€β”€ πŸ“‚result
 β”‚   └── πŸ“‚cifar10
 β”‚       └── πŸ“‚densenet201
 β”‚           └── πŸ“œdensenet201_cifar10_{corruption_type}_{severity_level}_result.pkl
 β”œβ”€β”€ ... 

Run using your own classifier

Place the network architecture files in the ./models/ directory and the pretrained weight files in the ./weights/ directory.

πŸ“¦Energy-Calibration
 β”œβ”€β”€πŸ“‚data
 β”‚  β”œβ”€β”€πŸ“‚cifar10
 β”‚  β”œβ”€β”€πŸ“‚cifar10C
 β”‚  β”œβ”€β”€πŸ“‚SVHN
 β”‚  β””β”€β”€πŸ“‚dtd
 β”œβ”€β”€πŸ“‚models             
 β”‚  β””β”€β”€πŸ“œ{network}.py    /* place your own network */
 β”œβ”€β”€πŸ“‚result
 β”œβ”€β”€πŸ“‚source
 β”œβ”€β”€πŸ“‚weights
 β”‚  β””β”€β”€πŸ“œ{network}.pth   /* place your own weight file */
 β”œβ”€β”€πŸ“œmain.py
 β”œβ”€β”€...

Run

python main.py --gpu --net {network} --weight_path {weight file path} 

Acknowledgements

This repository benefits from the following repositories:

We greatly appreciate their outstanding work and contributions to the community !

Citation

If you find this repository useful, please consider citing :

@inproceedings{kim2024uncertainty,
    title={Uncertainty Calibration with Energy Based Instance-wise Scaling in the Wild Dataset},
    author={Kim, Mijoo and Kwon, Junseok},
    booktitle={ECCV},
    year={2024}
}