Awesome
FourierHeatmap (latest release: v0.2.0)
This is an unofficial pytorch implementation of Fourier Heat Map which is proposed in the paper, A Fourier Perspective on Model Robustness in Computer Vision [Yin+, NeurIPS2019].
Fourier Heat Map allows to investigate the sensitivity of CNNs to high and low frequency corruptions via a perturbation analysis in the Fourier domain.
<img src="samples/FourierHeatmap-Teaser.png" height="300px">News
-
We release v0.2.0. API is renewed and some useful libraries (e.g. hydra) are added.
-
Previous version is still available as v0.1.0.
-
Docker is supported. Now, you can evaluate Fourier Heat Map on the Docker container.
Requirements
This library requires following as a pre-requisite.
- python 3.9+
- poetry
Note that I run the code with Ubuntu 20, Pytorch 1.8.1, CUDA 11.0.
Installation
This repo uses poetry as a package manager.
The following code will install all necessary libraries under .venv/
.
$ git clone git@github.com:gatheluck/FourierHeatmap.git
$ cd FourierHeatmap
$ pip install poetry # If you haven't installed poetry yet.
$ poetry install
Setup
Dataset
This codes expect datasets exist under data/
. For example, if you want to evaluate Fourier Heat Map for ImageNet, please set up like follows:
FourierHeatmap
├── data
│ └── imagenet
│ ├── train/
│ └── val/
Usage
Visualizing Fourier basis
The script fhmap/fourier/basis.py
generates Fourier base functions. For example:
$ poetry run python fhmap/fourier/basis.py
will generate 31x31 2D Fourier basis and save as an image under outputs/basis.png
. The generated image should be like follows.
Evaluating Fourier Heat Map
The script fhmap/apps/eval_fhmap.py
eveluate Fourier Heat Map for a model. For example:
$ poetry run python fhmap/apps/eval_fhmap.py dataset=cifar10 arch=resnet56 weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0
will generate 31x31 Fourier Heat Map for ResNet56 on CIFAR-10 dataset and save as an image under outputs/eval_fhmap/
. The generated image should be like follows.
Note that the L2 norm size (=eps) of Fourier basis use in original paper is following:
dataset | eps |
---|---|
CIFAR-10 | 4.0 |
ImageNet | 15.7 |
Evaluating custom dataset and model
Evaluating your custom dataset
If you want to evaluate Fourier Heat Map on your custom dataset, please refer follwing instraction.
-
Implement
YourCustomDatasetStats
class:- This class holds basic dataset information.
YourCustomDatasetStats
class should inherit from originalDatasetStats
class infactory/dataset
module and also shoud be placed infactory/dataset
module.- For details, please refer to the
Cifar10Stats
class infactory/dataset
module.
-
Implement
YourCustomDataModule
class:- This class is responsible for preprocess, transform (includes adding Fourier Noise to image) and create test dataset.
YourCustomDataModule
class should inherit fromBaseDataModule
class infactory/dataset
module and also shoud be placed infactory/dataset
module.- For details, please refer to the
Cifar10DataModule
class infactory/dataset
module.
-
Implement
YourCustomDatasetConfig
class:- This class is needed for applying hydra's dynamic object instantiation to dataset class.
YourCustomDatasetConfig
class should inherit fromDatasetConfig
class inschema/dataset
module and also shoud be placed inschema/dataset
module. Please addYourCustomDatasetConfig
toschema/__init__
.- For details, please refer to the
Cifar10Config
class inschema/dataset
module.
-
Add option for your custom dataset:
- Lastly, please add the config of your custom dataset to
ConfigStore
class by adding a follwing line toapps/eval_fhmap
.
cs.store(group="dataset", name="yourcustomdataset", node=schema.YourCustomDatasetConfig)
- Lastly, please add the config of your custom dataset to
Now, you will be able to call your custom dataset like following.
$ poetry run python fhmap/apps/eval_fhmap.py dataset=yourcustomdataset arch=resnet50 weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0
Evaluating your custom architecture (model)
If you want to evaluate Fourier Heat Map on your custom architecture (model), please refer follwing instraction.
-
Implement
YourCustomArch
class:- Please implement class or function which return your custom architecture. The custom architecture have to subclass of
torch.nn.module
. - For details, please refer to the
factory/archs/resnet
module.
- Please implement class or function which return your custom architecture. The custom architecture have to subclass of
-
Implement
YourCustomArchConfig
class:- This class is needed for applying hydra's dynamic object instantiation to architecture class.
YourCustomArchConfig
class should inherit fromArchConfig
class inschema/arch
module and also shoud be placed inschema/arch
module. Please addYourCustomArchConfig
toschema/__init__
.- For details, please refer to the
Resnet56Config
class inschema/arch
module. - If you want to use architectures which is provided by other libs like pytorch or timm, please refere to the
Resnet50Config
class inschema/arch
module.
-
Add option for your custom architecture:
- Lastly, please add the config of your custom architecture to
ConfigStore
class by adding a follwing line toapps/eval_fhmap
.
cs.store(group="arch", name="yourcustomarch", node=schema.YourCustomArchConfig)
- Lastly, please add the config of your custom architecture to
Now, you will be able to call your custom arch like following.
$ poetry run python fhmap/apps/eval_fhmap.py dataset=cifar10 arch=yourcustomarch weightpath=[PYTORCH_MODEL_WEIGHT_PATH] eps=4.0
Evaluating Fourier Heat Map through Docker
In order to use FourierHeatmap throgh docker, please install Docker with NVIDIA Container Toolkit beforehand. For detail, please refere official installation guide.
If nvidia-smi
is able to run through docker like following, it is successfully installed.
$ sudo docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi
Tue Apr 27 06:46:09 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.102.04 Driver Version: 450.102.04 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce GTX 1080 Off | 00000000:01:00.0 On | N/A |
| N/A 56C P0 42W / N/A | 1809MiB / 8114MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
We use environmental variables to specify the arguments. The variables that can be specified and their meanings are as follows:
name | optional | default | description |
---|---|---|---|
HOST_DATADIR | False | Path to the directory where the dataset is located in the host. | |
HOST_OUTPUTSDIR | False | Path to the directory where the output will be located in the host. | |
HOST_WEIGHTDIR | False | Path to the directory where the pretrained wight is located in the host. | |
WEIGHTFILE | False | File name of the pretrained wight. | |
ARCH | True | resnet56 | Name of the architecture. |
BATCH_SIZE | True | 512 | Size of batch. |
DATASET | True | cifar10 | Name of dataset. |
EPS | True | 4.0 | L2 norm size of Fourier basis. |
IGNORE_EDGE_SIZE | True | 0 | Size of the edge to ignore. |
NUM_SAMPLES | True | -1 | Number of samples used from dataset. If -1, use all samples. |
NVIDIA_VISIBLE_DEVICES | True | 0 | Device number (or list of number) visible from CUDA. |
For example:
$ export HOST_DATADIR=[DATASET_DIRECTORY_PATH]
$ export HOST_OUTPUTSDIR=[OUTPUTS_DIRECTORY_PATH]
$ export HOST_WEIGHTDIR=[WEIGHT_DIRECTORY_PATH]
$ export WEIGHTFILE=[PYTORCH_MODEL_FILE]
$ cd provision/docker
$ sudo -E docker-compose up # -E option is needed to inherit environment variables.
will generate 31x31 Fourier Heat Map for ResNet56 on CIFAR-10 dataset and save as an image under OUTPUTS_DIRECTORY_PATH/eval_fhmap/
.