Home

Awesome

Convolutional Neural Networks for CIFAR-10

This repository is about some implementations of CNN Architecture for cifar10.

cifar10

I just use Keras and Tensorflow to implementate all of these CNN models.
(maybe torch/pytorch version if I have time)
A pytorch version is available at CIFAR-ZOO

Requirements

Architectures and papers

Documents & tutorials

There are also some documents and tutorials in doc & issues/3.
Get it if you need.
You can also see the articles if you can speak Chinese. <img src="https://user-images.githubusercontent.com/7837172/44953504-b9481000-aec8-11e8-9920-abf66365b8d8.gif">

Accuracy of all my implementations

In particular
Change the batch size according to your GPU's memory.
Modify the learning rate schedule may imporve the results of accuracy!

networkGPUparamsbatch sizeepochtraining timeaccuracy(%)
Lecun-NetworkGTX1080TI62k12820030 min76.23
Network-in-NetworkGTX1080TI0.97M1282001 h 40 min91.63
Vgg19-NetworkGTX1080TI39M1282001 h 53 min93.53
Residual-Network20GTX1080TI0.27M12820044 min91.82
Residual-Network32GTX1080TI0.47M1282001 h 7 min92.68
Residual-Network110GTX1080TI1.7M1282003 h 38 min93.93
Wide-resnet 16x8GTX1080TI11.3M1282004 h 55 min95.13
Wide-resnet 28x10GTX1080TI36.5M12820010 h 22 min95.78
DenseNet-100x12GTX1080TI0.85M6425017 h 20 min94.91
DenseNet-100x24GTX1080TI3.3M6425022 h 27 min95.30
DenseNet-160x241080 x 27.5M6425050 h 20 min95.90
ResNeXt-4x64dGTX1080TI20M12025021 h 3 min95.19
SENet(ResNeXt-4x64d)GTX1080TI20M12025021 h 57 min95.60

About LeNet and CNN training tips/tricks

LeNet is the first CNN network proposed by LeCun.
I used different CNN training tricks to show you how to train your model efficiently.

LeNet_keras.py is the baseline of LeNet,
LeNet_dp_keras.py used the Data Prepossessing [DP],
LeNet_dp_da_keras.py used both DP and the Data Augmentation[DA],
LeNet_dp_da_wd_keras.py used DP, DA and Weight Decay [WD]

networkGPUDPDAWDtraining timeaccuracy(%)
LeNet_kerasGTX1080TI---5 min58.48
LeNet_dp_kerasGTX1080TI--5 min60.41
LeNet_dp_da_kerasGTX1080TI-26 min75.06
LeNet_dp_da_wd_kerasGTX1080TI26 min76.23

For more CNN training tricks, see Must Know Tips/Tricks in Deep Neural Networks (by Xiu-Shen Wei)

About Learning Rate schedule

Different learning rate schedule may get different training/testing accuracy!
See ./htd, and HTD for more details.

About Multiple GPUs Training

Since the latest version of Keras is already supported keras.utils.multi_gpu_model, so you can simply use the following code to train your model with multiple GPUs:

from keras.utils import multi_gpu_model
from keras.applications.resnet50 import ResNet50

model = ResNet50()

# Replicates `model` on 8 GPUs.
parallel_model = multi_gpu_model(model, gpus=8)
parallel_model.compile(loss='categorical_crossentropy',optimizer='adam')

# This `fit` call will be distributed on 8 GPUs.
# Since the batch size is 256, each GPU will process 32 samples.
parallel_model.fit(x, y, epochs=20, batch_size=256)

About ResNeXt & DenseNet

Since I don't have enough machines to train the larger networks, I only trained the smallest network described in the paper. You can see the results in liuzhuang13/DenseNet and prlz77/ResNeXt.pytorch

<a href="https://bigballon.github.io"> <img src="https://user-images.githubusercontent.com/7837172/44953504-b9481000-aec8-11e8-9920-abf66365b8d8.gif"></a> <a href="https://bigballon.github.io"> <img src="https://user-images.githubusercontent.com/7837172/44953504-b9481000-aec8-11e8-9920-abf66365b8d8.gif"></a>

Please feel free to contact me if you have any questions!

Citation

@misc{bigballon2017cifar10cnn,
  author = {Wei Li},
  title = {cifar-10-cnn: Play deep learning with CIFAR datasets},
  howpublished = {\url{https://github.com/BIGBALLON/cifar-10-cnn}},
  year = {2017}
}