Awesome
Wide Residual Networks Using Ensemble
Wide-residual network implementations for cifar10, cifar100, and other kaggle challenges
Torch Implementation of Sergey Zagoruyko's Wide Residual Networks.
In order to figure out what 'width' & 'height' does on wide-residual networks, several experiments were conducted on different settings of weights and heights. It turns out that increasing the number of filters(increasing width) gave more positive influence to the model than making the model deeper.
Last but not least, simply averaging a few models with different parameter settings showed a significant increase in both top1 and top5 accuracy. The CIFAR dataset test results approached to 97.12% for CIFAR-10, and 84.19% for CIFAR-100 with only meanstd normalization.
Requirements
See the installation instruction for a step-by-step installation guide. See the server instruction for server setup.
- Install Torch
- Install cuda-8.0
- Install cudnn v5.1
- Install luarocks packages
$ luarocks install cutorch
$ luarocks install xlua
$ luarocks install optnet
Directions and datasets
- modelState : The best model will be saved in this directory
- datasets : Data preparation & preprocessing directory
- networks : Wide-residual network model structure file directory
- gen : Generated t7 file for each dataset will be saved in this directory
- scripts : Directory where the run file scripts are contained
Best Results
CIFAR-10's top1 accuracy reaches to 97.12% only with average ensembling without any weight adjustments.
Adapting weight adjustments for each model will promise a more improved accuracy.
You can see that the ensemble network improves the results of single WRNs.
Test error (%, random flip, meanstd normaliztion, median of 5 runs) on CIFAR:
| Dataset | network | Top1 Err(%) |
|---|---|---|
| CIFAR-10 | WRN-28x10 | 3.89 |
| CIFAR-10 | Ensemble-WRN | 2.88 |
| CIFAR-100 | WRN-28x10 | 18.85 |
| CIFAR-100 | Ensemble-WRN | 15.81 |
How to run
You can train each dataset of either cifar10, cifar100 or svhn by running the script below.
$ ./scripts/[:dataset]_train.sh
# For example, if you want to train the model on cifar10, you simply type
$ ./scripts/cifar10_train.sh
You can test your own trained model of either cifar10, cifar100, svhn by running the script below.
$ ./scripts/[:dataset]_test.sh
To ensemble your multiple trained models of different parameters, follow the steps below.
$ vi ensemble.lua
# Press :32 in vi, which will move your cursor to line 32
ens_depth = torch.Tensor({28, 28, 28, 28, 40, 40, 40})
ens_widen_factor = torch.Tensor({20, 20, 20, 20, 10, 14, 14})
ens_nExperiment = torch.Tensor({ 2, 3, 4, 5, 5, 4, 5})
After you set each parameter for your models, open scripts/ensemble.sh
$ vi scripts/ensemble.sh
# on the second line
export dataset=[:dataset] # put the dataset you want to ensemble your models.
export mode=[:mode] # you can either choose 'avg', 'min', 'max'
Finally, run the script file.
$ ./scripts/ensemble.sh
Implementation Details
- CIFAR-10, CIFAR-100
| epoch | learning rate | weight decay | Optimizer | Momentum | Nesterov |
|---|---|---|---|---|---|
| 0 ~ 60 | 0.1 | 0.0005 | Momentum | 0.9 | true |
| 61 ~ 120 | 0.02 | 0.0005 | Momentum | 0.9 | true |
| 121 ~ 160 | 0.004 | 0.0005 | Momentum | 0.9 | true |
| 161 ~ 200 | 0.0008 | 0.0005 | Momentum | 0.9 | true |
- SVHN
| epoch | learning rate | weight decay | Optimizer | Momentum | Nesterov |
|---|---|---|---|---|---|
| 0 ~ 80 | 0.01 | 0.0005 | Momentum | 0.9 | true |
| 81 ~ 120 | 0.001 | 0.0005 | Momentum | 0.9 | true |
| 121 ~ 160 | 0.0001 | 0.0005 | Momentum | 0.9 | true |
CIFAR-10 Results
Below is the result of the test set accuracy for CIFAR-10 dataset training.
Accuracy is the average of 5 runs
| network | dropout | preprocess | GPU:0 | GPU:1 | per epoch | accuracy(%) |
|---|---|---|---|---|---|---|
| pre-ResNet-1001 | 0 | meanstd | - | - | 3 min 25 sec | 95.08 |
| wide-resnet 28x10 | 0 | ZCA | 5.90G | - | 2 min 03 sec | 95.84 |
| wide-resnet 28x10 | 0 | meanstd | 5.90G | - | 2 min 03 sec | 96.01 |
| wide-resnet 28x10 | 0.3 | meanstd | 5.90G | - | 2 min 03 sec | 96.19 |
| wide-resnet 28x20 | 0.3 | meanstd | 8.13G | 6.93G | 4 min 10 sec | 96.52 |
| wide-resnet 40x10 | 0.3 | meanstd | 8.08G | - | 3 min 13 sec | 96.26 |
| wide-resnet 40x14 | 0.3 | meanstd | 7.37G | 6.46G | 3 min 23 sec | 96.31 |
CIFAR-100 Results
Below is the result of the test set accuracy for CIFAR-100 dataset training.
Accuracy is the average of 5 runs
| network | dropout | preprocess | GPU:0 | GPU:1 | per epoch | Top1 acc(%) | Top5 acc(%) |
|---|---|---|---|---|---|---|---|
| pre-ResNet-1001 | 0 | meanstd | - | - | 3 min 25 sec | 77.29 | 93.44 |
| wide-resnet 28x10 | 0 | ZCA | 5.90G | - | 2 min 03 sec | 80.03 | 95.01 |
| wide-resnet 28x10 | 0 | meanstd | 5.90G | - | 2 min 03 sec | 81.01 | 95.44 |
| wide-resnet 28x10 | 0.3 | meanstd | 5.90G | - | 2 min 03 sec | 81.47 | 95.53 |
| wide-resnet 28x20 | 0.3 | meanstd | 8.13G | 6.93G | 4 min 05 sec | 82.43 | 96.02 |
| wide-resnet 40x10 | 0.3 | meanstd | 8.93G | - | 3 min 06 sec | 81.47 | 95.65 |
| wide-resnet 40x14 | 0.3 | meanstd | 7.39G | 6.46G | 3 min 23 sec | 81.83 | 95.50 |
SVHN Results
Below is the result of the test set accrucay for SVHN dataset training.
Accuracy is the result of 1 run
| network | dropout | preprocess | GPU:0 | per epoch | Top1 acc(%) |
|---|---|---|---|---|---|
| wide-resnet 10x1 | 0.4 | meanstd | 0.91G | 1 min 37 sec | 93.815 |
| wide-resnet 10x8 | 0.4 | meanstd | 2.03G | 7 min 32 sec | 97.411 |
| wide-resnet 16x8 | 0.4 | meanstd | 2.92G | 14 min 8 sec | 98.229 |
| wide-resnet 22x8 | 0.4 | meanstd | 3.73G | 21 min 11 sec | 98.348 |