Home

Awesome

Creating BREEDS Sub-population shift Benchmarks

This repository contains the code and data for our paper:

BREEDS: Benchmarks for Subpopulation Shift <br> Shibani Santurkar*, Dimitris Tsipras*, Aleksander Madry <br> Paper: https://arxiv.org/abs/2008.04859 <br>

    @InProceedings{santurkar2020breeds,
        title={BREEDS: Benchmarks for Subpopulation Shift},
        author={Shibani Santurkar and Dimitris Tsipras and Aleksander Madry},
        year={2020},
        booktitle={ArXiv preprint arXiv:2008.04859}
    }

Getting started

Our code relies on the MadryLab public robustness library, which will be automatically installed when you follow the instructions below.

  1. Clone our repo: git clone https://github.com/MadryLab/BREEDS-Benchmarks.git

  2. Install dependencies:

    conda create -n breeds-benchmarks python=3.7 pip
conda activate breeds-benchmarks
pip install -r requirements.txt
conda install pygraphviz

  3. Download the ImageNet dataset.

That's it! Now you can create new BREEDS datasets, or look closer at the ones we use in our paper using this notebook. You can also find a detailed walkthrough of this code in this documentation. In order to train models using these datasets, you can use simply use existing code from the robustness library.

Maintainers