Awesome
Outlier-Aware Test-Time Adaptation with Stable Memory Replay
[paper]
Prerequisites
To use the repository, we provide a conda environment.
conda update conda
conda env create -f environment.yaml
conda activate stamp
Structure of Project
This project is based on a TTA-Benchmark containing several directories. Their roles are listed as follows:
- ./cfgs: the config files for each dataset and algorithm are saved here.
- ./robustbench: an official library we use to load robust datasets and models.
- ./src/
- data: we load our datasets and dataloaders by code under this directory.
- methods: the code for the implementation of various TTA methods.
- models: the various models' loading process and definition rely on the code here.
- utils: some useful tools for our projects.
Run
This repository allows to study a wide range of different datasets, models, settings, and methods. A quick overview is given below:
-
Datasets
-
cifar10_c
CIFAR10-C -
cifar100_c
CIFAR100-C -
imagenet_c
ImageNet-C -
LSUN-C
LSUN -
SVHN-C
SVHN -
Tiny-ImageNet-C
Tiny-ImageNet-C -
Textures-C
Textures -
Places365-C
Places365
-
The dataset directory structure is as follows:
|-- datasets
|-- cifar-10
|-- cifar-100
|-- ImageNet
|-- train
|-- val
|-- ImageNet-C
|-- CIFAR-10-C
|-- CIFAR-100-C
|-- LSUN_resize-C
|-- PLACES365-C
|-- SVHN-C
|-- Textures-C
|-- Tiny-ImageNet-C
For OOD datasets, you can generate the corrupted datasets according to the instructions in this repository or robustbench.
-
Models
-
You can train the source model by script in the ./pretrain directory.
-
You can also download our checkpoint from here.
-
-
Methods
-
Modular Design
- Adding new methods should be rather simple, thanks to the modular design.
Get Started
To run one of the following benchmarks, the corresponding datasets need to be downloaded.
Next, specify the root folder for all datasets _C.DATA_DIR = "./data"
in the file conf.py
.
download the checkpoints of pre-trained models from here and put it in ./ckpt
How to reproduce
The entry file for algorithms is test-time-eva-baseline.sh
To evaluate these methods, modify the DATASET and METHOD in test-time-eva.sh
and then
bash test-time-eva-baseline.sh