Awesome

Official Benchmark Implementation

:scroll: A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification

The original paper establishing the FD-Shifts benchmark was presented as an Oral at ICLR 2023 (top 5%).

→ project page → paper link

:scroll: Understanding Silent Failures in Medical Image Classification

Our follow-up study on Failure Detection in Medical Image Classification was presented at MICCAI 2023.

→ project page → paper link → interactive tool SF-Visuals

:scroll: Overcoming Common Flaws in the Evaluation of Selective Classification Systems

Our paper on a revised evaluation protocol for Selective Classification Systems was accepted as Spotlight paper at NeurIPS 2024.

→ project page → paper link → AUGRC implementation

Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores. This confidence may be obtained by either quantifying the model's predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution. Curiously, while these approaches all state to address the same eventual goal of detecting failures of a classifier upon real-life application, they currently constitute largely separated research fields with individual evaluation protocols, which either exclude a substantial part of relevant methods or ignore large parts of relevant failure sources. In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for a holistic and realistic evaluation of failure detection. To demonstrate the relevance of this unified perspective, we present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t all relevant methods and failure sources. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the abundance of publicized research on confidence scoring.

Citing This Work

If you use FD-Shifts please cite our paper

@inproceedings{
    jaeger2023a,
    title={A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification},
    author={Paul F Jaeger and Carsten Tim L{\"u}th and Lukas Klein and Till J. Bungert},
    booktitle={International Conference on Learning Representations},
    year={2023},
    url={https://openreview.net/forum?id=YnkGMIh0gvX}
}

Table Of Contents

• Citing This Work
• Table Of Contents
• Installation
• How to Integrate Your Own Usecase
• Reproducing our results
• Working with FD-Shifts
  • Data Folder Requirements
  • Training
  • Model Weights
  • Inference
  • Analysis
• Acknowledgements

Installation

FD-Shifts requires Python version 3.10 or later. It is recommended to install FD-Shifts in its own environment (venv, conda environment, ...).

1. Install an appropriate version of PyTorch. Check that CUDA is available and that the CUDA toolkit version is compatible with your hardware. The currently minimum necessary version of pytorch is v.1.11.0. Testing and Development was done with the pytorch version using CUDA 11.3.

2. Install FD-Shifts. This will pull in all dependencies including some version of PyTorch, it is strongly recommended that you install a compatible version of PyTorch beforehand. This will also make the fd-shifts cli available to you.

   pip install git+https://github.com/iml-dkfz/fd-shifts.git
   

How to Integrate Your Own Usecase

To learn about extending FD-Shifts with your own models, datasets and confidence scoring functions check out the tutorial on extending FD-Shifts .

Reproducing our results

While the following section on working with FD-Shifts describes the general usage, descriptions for reproducing specific publications are documented on the respective project page:

• "A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification"
• "Understanding Silent Failures in Medical Image Classification"
• "Overcoming Common Flaws in the Evaluation of Selective Classification Systems"

Working with FD-Shifts

To use fd-shifts you need to set the following environment variables

export EXPERIMENT_ROOT_DIR=/absolute/path/to/your/experiments
export DATASET_ROOT_DIR=/absolute/path/to/datasets

Alternatively, you may write them to a file and source that before running fd-shifts, e.g.

mv example.env .env

Then edit .env to your needs and run

source .env

To get an overview of available subcommands, run fd-shifts --help.

Data Folder Requirements

For the predefined experiments we expect the data to be in the following folder structure relative to the folder you set for $DATASET_ROOT_DIR.

<$DATASET_ROOT_DIR>
├── breeds
│   └── ILSVRC ⇒ ../imagenet/ILSVRC
├── imagenet
│   ├── ILSVRC
├── cifar10
├── cifar100
├── corrupt_cifar10
├── corrupt_cifar100
├── svhn
├── tinyimagenet
├── tinyimagenet_resize
├── wilds_animals
│   └── iwildcam_v2.0
└── wilds_camelyon
    └── camelyon17_v1.0

For information regarding where to download these datasets from and what you have to do with them please check out the dataset documentation.

Training

To get a list of all fully qualified names for all experiments in the paper, use

fd-shifts list-experiments

To run training for a specific experiment:

fd-shifts train --experiment=svhn_paper_sweep/devries_bbsvhn_small_conv_do1_run1_rew2.2

Alternatively, run training from a custom configuration file:

fd-shifts train --config=path/to/config/file

Check out fd-shifts train --help for more training options.

The launch subcommand allows for running multiple experiments, e.g. filtered by dataset:

fd-shifts launch --mode=train --dataset=cifar10

Check out fd-shifts launch --help for more filtering options. You can add custom experiment filters via the register_filter decorator. See experiments/launcher.py for an example.

Model Weights

All pretrained model weights used for "A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification" can be found on Zenodo under the following links:

• iWildCam-2020-Wilds
• iWildCam-2020-Wilds (OpenSet Training)
• BREEDS-ENTITY-13
• CAMELYON-17-Wilds
• CIFAR-100
• CIFAR-100 (superclasses)
• CIFAR-10
• SVHN
• SVHN (OpenSet Training)

Inference

To run inference for one of the experiments:

fd-shifts test --experiment=svhn_paper_sweep/devries_bbsvhn_small_conv_do1_run1_rew2.2

Analogously, with the launch subcommand:

fd-shifts launch --mode=test --dataset=cifar10

Analysis

To run analysis for one of the experiments:

fd-shifts analysis --experiment=svhn_paper_sweep/devries_bbsvhn_small_conv_do1_run1_rew2.2

To run analysis over an already available set of inference outputs the outputs have to be in the following format:

For a classifier with d outputs and N samples in total (over all tested datasets) and for M dropout samples

raw_logits.npz
Nx(d+2)

  0, 1, ...                 d─1,   d,      d+1
┌───────────────────────────────┬───────┬─────────────┐
|           logits_1            | label | dataset_idx |
├───────────────────────────────┼───────┼─────────────┤
|           logits_2            | label | dataset_idx |
├───────────────────────────────┼───────┼─────────────┤
|           logits_3            | label | dataset_idx |
└───────────────────────────────┴───────┴─────────────┘
.
.
.
┌───────────────────────────────┬───────┬─────────────┐
|           logits_N            | label | dataset_idx |
└───────────────────────────────┴───────┴─────────────┘

external_confids.npz
Nx1

raw_logits_dist.npz
NxdxM

  0, 1, ...                  d─1
┌───────────────────────────────┐
|   logits_1 (Dropout Sample 1) |
|   logits_1 (Dropout Sample 2) |
|               .               |
|               .               |
|               .               |
|   logits_1 (Dropout Sample M) |
├───────────────────────────────┤
|   logits_2 (Dropout Sample 1) |
|   logits_2 (Dropout Sample 2) |
|               .               |
|               .               |
|               .               |
|   logits_2 (Dropout Sample M) |
├───────────────────────────────┤
|   logits_3 (Dropout Sample 1) |
|   logits_3 (Dropout Sample 2) |
|               .               |
|               .               |
|               .               |
|   logits_3 (Dropout Sample M) |
└───────────────────────────────┘
                .
                .
                .
┌───────────────────────────────┐
|   logits_N (Dropout Sample 1) |
|   logits_N (Dropout Sample 2) |
|               .               |
|               .               |
|               .               |
|   logits_N (Dropout Sample M) |
└───────────────────────────────┘

external_confids_dist.npz
NxM

To load inference output from different locations than $EXPERIMENT_ROOT_DIR, you can specify one or multiple directories in the FD_SHIFTS_STORE_PATH environment variable (multiple paths are separated by :):

export FD_SHIFTS_STORE_PATH=/absolute/path/to/fd-shifts/inference/output

You may also use the ExperimentData class to load your data in another way. You also have to provide an adequate config, where all test datasets and query parameters are set. Check out the config files in fd_shifts/configs including the dataclasses. Importantly, the dataset_idx has to match up with the list of datasets you provide and whether or not val_tuning is set. If val_tuning is set, the validation set takes over dataset_idx=0.

Acknowledgements