Home

Awesome

ECCV2024-Dataset-Distillation-Challenge

This repository contains the evaluation code for the Data Compression Challenge. The code evaluates the performance of submitted models on CIFAR-100 and Tiny ImageNet datasets. The evaluation is performed on an NVidia 4090

The sample submission can be downloaded sample_submission.zip.

The Test Set of CIFAR-100 and Tiny-ImageNet can be downloaded reference_data.zip.

Important Design Notes

Please keep in mind, that the test data is normalized following the standard normalization technqiues for CIFAR100 and TinyImagenet. In particular we assume your distilled data has been learned from a normalized training dataset using:

#* CIFAR100
# mean = [0.5071, 0.4866, 0.4409]
# std = [0.2673, 0.2564, 0.2762]

#* TinyImagenet
# mean = [0.485, 0.456, 0.406]
# std = [0.229, 0.224, 0.225]

We do not perform normalization in this evaluation script -- your data must be pre-normalized (i.e normalized before the distillation following other commone distillation works). The sample submission data represents a random selection at IPC10 for both CIFAR100 and TinyImagenet.

Usage (Track 1)

  1. Please follow the same heiarchial structure as our sample_submission.

  2. Please unzip the reference testing data "reference_data.zip" and create the folder structure "./reference_data/{cifar100|tinyimagenet}_test.pt"

  3. Please unzip the sample submission data "sample_submission.zip" and create the folder structure "./sample_submission/{cifar100|tinyimagenet}.pt". Note: "sample_submission" contains 2 files: "cifar100.pt" and "tinyimagenet.pt". Both files contain randomly samples images from the respective datasets at IPC 10. Please follow the same structure when creating and saving your distilled data.

  4. To evaluate your data, please set the "--submit_dir {your_path}"

python evaluate.py --submit_dir {path-to-your-data}

Alternatively, to evaluate the sample use:

python evaluate.py --submit_dir ./sample_submission/

Script Explanation

Troubleshooting