Awesome
IterNorm-pytorch
Pytorch reimplementation of the IterNorm methods, which is described in the following paper:
Iterative Normalization: Beyond Standardization towards Efficient Whitening
Lei Huang, Yi Zhou, Fan Zhu, Li Liu, Ling Shao
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019 (accepted). arXiv:1904.03441
This project also provide the pytorch implementation of Decorrelated Batch Normalization (CVPR 2018, arXiv:1804.08450), more details please refer to the Torch project.
Requirements and Dependency
- Install PyTorch with CUDA (for GPU). (Experiments are validated on python 3.6.8 and pytorch-nightly 1.0.0)
- (For visualization if needed), install the dependency visdom by:
pip install visdom
Experiments
1. VGG-network on Cifar-10 datasets:
run the scripts in the ./cifar10/experiments/vgg
. Note that the dataset root dir should be altered by setting the para '--dataset-root', and the dataset style is described as:
-<dataset-root>
|-cifar10-batches-py
||-data_batch_1
||-data_batch_2
||-data_batch_3
||-data_batch_4
||-data_batch_5
||-test_batch
If the dataset is not exist, the script will download it, under the conditioning that the dataset-root
dir is existed
2. Wide-Residual-Network on Cifar-10 datasets:
run the scripts in the ./cifar10/experiments/wrn
.
3. ImageNet experiments.
run the scripts in the ./ImageNet/experiment
. Note that resnet18 experimetns are run on one GPU, and resnet-50/101 are run on 4 GPU in the scripts.
Note that the dataset root dir should be altered by setting the para '--dataset-root'. and the dataset style is described as:
-<dataset-root>
|-train
||-class1
||-...
||-class1000
|-var
||-class1
||-...
||-class1000
Using IterNorm in other projects/tasks
(1) copy ./extension/normalization/iterative_normalization.py
to the respective dir.
(2) import the IterNorm
class in iterative_normalization.py
(3) generally speaking, replace the BatchNorm
layer by IterNorm
, or add it in any place if you want to the feature/channel decorrelated. Considering the efficiency (Note that BatchNorm
is intergrated in cudnn
while IterNorm
is based on the pytorch script without optimization), we recommend 1) replace the first BatchNorm
; 2) insert extra IterNorm
before the first skip connection in resnet; 3) inserted before the final linear classfier as described in the paper.
(4) Some tips related to the hyperparamters (Group size G
and Iterative Number T
). We recommend G=64
(i.e., the channel number in per group is 64) and T=5
by default. If you run on large batch size (e.g.>1024), you can either increase G
or T
. For fine tunning, fix G=64 or G=32
, and search T={3,4,5,6,7,8}
may help.