Home

Awesome

CIFAR-10.1

This repository contains the CIFAR-10.1 dataset, which is a new test set for CIFAR-10.

CIFAR-10.1 contains roughly 2,000 new test images that were sampled after multiple years of research on the original CIFAR-10 dataset. The data collection for CIFAR-10.1 was designed to minimize distribution shift relative to the original dataset. We describe the creation of CIFAR-10.1 in the paper "Do CIFAR-10 Classifiers Generalize to CIFAR-10?". The images in CIFAR-10.1 are a subset of the TinyImages dataset.

Using the Dataset

Dataset Releases

There are currently two versions of the CIFAR-10.1 dataset:

The overlap between v4 and v6 is more than 90% of the respective datasets. Moreover, the classification accuracies are very close (see Appendix D of our paper). For future experiments, we recommend the v6 version of our dataset.

Missing version numbers correspond to internal releases during our quality control process (e.g., near-duplicate removal) or potential variants of our dataset we did not pursue further.

Loading the Dataset

The datasets directory contains the dataset files in the NumPy binary format:

The notebooks directory contains a short script inspect_dataset_simple.ipynb to browse the CIFAR-10.1 dataset. The notebook uses a utility function to load the dataset from utils.py in the the code directory.

Dataset Creation Pipeline

WARNING: This is currently work in progress, some parts may be incomplete.

This repository contains code to replicate the creation process of CIFAR-10.1. The dataset creation process has several stages outlined below. We describe the process here at a high level. If you have questions about any individual steps, please do not hesitate to contact Rebecca Roelofs (roelofs@cs.berkeley.edu) and Ludwig Schmidt (ludwigschmidt2@gmail.com).

1. Extracting Data from TinyImages

Since the TinyImages dataset is quite large (around 280 GB), we first extract the relevant data for further processing. In particular, we require the following information:

We have automated these two steps via two scripts in the code directory:

We recommend running these scripts on a machine with at least 1 TB of RAM, e.g., an x1.16xlarge instance on AWS. After downloading the TinyImage dataset, running the scripts will take about 30h.

The scripts will produce the following data files, all of which are stored in the other_data folder:

2. Collecting Candidate Images

After downloading the relevant subset of TinyImages (keywords and image data) to a local machine, we can now assemble a set of candidate images for the new dataset. We proceed in two steps:

2.1 Keyword counts for the new dataset

The notebook generate_keyword_counts.ipynb decides which keywords we want to include in the new dataset and determines the number of images we require for each of these keywords.

2.2 Labeling new images

Once we know the number of new images we require for each keyword, we can collect corresponding images from TinyImages. We used two notebooks for this process:

3. Assembling a New Dataset

Given a pool of new candidate images, we can now sample a new dataset from this pool. We have the following notebooks for this step:

After sampling a new dataset, it is necessary to run some final checks via the check_dataset_ui.ipynb notebook. In particular, this notebook checks for near-duplicates both within the new test set and in CIFAR-10 (a new test set would not be interesting if it contains many near-duplicates of the original test set). In our experience, the process involves a few round-trips of sampling a new test set, checking for near-duplicates, and adding the near-duplicates to the blacklist. Sometimes it is necessary to collect a few additional images for keywords with many near-duplicates (using the notebooks from Step 2 above).

In order to avoid re-computing L2 distances to CIFAR-10, the notebook compute_distances_to_cifar10.ipynb computes all top-10 nearest neighbors between our TinyImages subset and CIFAR-10. Running this notebook takes only a few minutes when executed on 100 m5.4xlarge instances via PyWren.

4. Inspecting Model Predictions (Extra Step)

After assembling a final dataset, we ran a broad range of classifiers on the new test set via our CIFAR-10 model test bed. The notebook inspect_model_predictions.ipynb explores the resulting predictions and displays a Pandas dataframe including the original and new accuracy for each model.

Intermediate Data Files

In order to run only individual steps of the process outlined above, we provide all intermediate data files. They are stored in the S3 bucket cifar-10-1 and can be downloaded with the script other_data/download.py. The script requires Boto 3, which can be installed via pip: pip install boto3.

License

Unless noted otherwise in individual files, the code in this repository is released under the MIT license (see the LICENSE file). The LICENSE file does not apply to the actual image and label data in the datasets folder. The image data is part of the Tiny Images dataset and can be used the same way as the Tiny Images dataset.

Citing CIFAR-10.1

To cite the CIFAR-10.1 dataset, please use the following references:

@article{recht2018cifar10.1,
  author = {Benjamin Recht and Rebecca Roelofs and Ludwig Schmidt and Vaishaal Shankar},
  title = {Do CIFAR-10 Classifiers Generalize to CIFAR-10?},
  year = {2018},
  note = {\url{https://arxiv.org/abs/1806.00451}},
}

@article{torralba2008tinyimages, 
  author = {Antonio Torralba and Rob Fergus and William T. Freeman}, 
  journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title = {80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition}, 
  year = {2008}, 
  volume = {30}, 
  number = {11}, 
  pages = {1958-1970}
}