Home

Awesome

G-CNN Experiments

Code for reproducing the experiments reported in: T.S. Cohen, M. Welling, Group Equivariant Convolutional Networks. Proceedings of the International Conference on Machine Learning (ICML), 2016

p4_anim

A rotating feature map on the rotation-translation group p4. See section 4 of the paper.

Results

Comparison to other methods

There are two common regimes for comparing classifiers on CIFAR10:

NetworkCIFAR10CIFAR10+
Maxout [8]11.689.38
DropConnect [9]9.32
NiN [10]10.418.81
DSN [13]9.697.97
All-CNN-C [11]9.077.25
Highway nets [15]7.6
ELU [12]6.55
Generalized Pooling [14]7.626.05
ResNet1001 [18]4.62
ResNet101 [17]6.43
Wide ResNet 28 [16]4.17
p4m-Resnet26 (ours)5.744.19

Comparison of G-CNNs

We compare the following group convolutions:

We compare the following architectures:

NetworkGCIFAR10CIFAR10+Param.
All-CNN-CZ<sup>2</sup>9.448.861.37M
p48.847.671.37M
p4m7.597.041.22M
ResNet44Z<sup>2</sup>9.455.612.64M
p4m6.464.942.62M
ResNet26Z<sup>2</sup>8.955.277.24M
p4m5.744.197.17M

We see a very consistent behaviour:

Installation

Install scientific python stack and progressbar

$ pip install ipython numpy scipy matplotlib progressbar2 skimage

Install chainer with CUDNN and HDF5: installation instructions

Install GrouPy

Add the gconv_experiments folder to your PYTHONPATH.

Download data

CIFAR10

$ cd [datadir]
$ wget http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
$ tar zxvf cifar-10-python.tar.gz
$ rm cifar-10-python.tar.gz

MNIST-rot

$ cd [datadir]
$ wget http://www.iro.umontreal.ca/~lisa/icml2007data/mnist_rotation_new.zip
$ unzip mnist_rotation_new.zip 
$ rm mnist_rotation_new.zip
$ ipython /path/to/gconv_experiments/gconv_experiments/MNIST_ROT/mnist_rot.py -- --datadir=./

Train a G-CNN

MNIST-rot

To run the MNIST-rot experiments:

$ ipython MNIST_ROT/experiment.py -- --trainfn=[datadir]/train_all.npz --valfn=[datadir]/test.npz

You can also call train.py directly to train a single model.

CIFAR10

The first time you run an experiment, the code will preprocess the dataset and leave a preprocessed copy in [datadir].

$ cd gconv_experiments
$ ipython CIFAR10/train.py -- --datadir=[datadir] --resultdir=[resultdir] --modelfn=CIFAR10/models/P4AllCNNC.py

For other options, see train.py.

References

Related work

  1. Kivinen, Jyri J. and Williams, Christopher K I. Transformation equivariant Boltzmann machines. In 21st International Conference on Artificial Neural Networks, 2011.
  2. Sifre, Laurent and Mallat, Stephane. Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination. IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2013.
  3. Gens, R. and Domingos, P. Deep Symmetry Networks. In Advances in Neural Information Processing Systems (NIPS), 2014.
  4. Jaderberg, M., Simonyan, K., Zisserman, A., and Kavukcuoglu, K. Spatial Transformer Networks. In Advances in Neural Information Processing Systems 28 (NIPS 2015), 2015.
  5. Oyallon, E. and Mallat, S. Deep Roto-Translation Scattering for Object Classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
  6. Zhang, C., Voinea, S., Evangelopoulos, G., Rosasco, L., and Poggio, T. Discriminative template learning in group-convolutional networks for invariant speech representations. InterSpeech, 2015.
  7. Dieleman, S., De Fauw, J., and Kavukcuoglu, K. Exploiting Cyclic Symmetry in Convolutional Neural Networks. In International Conference on Machine Learning (ICML), 2016.

Papers reporting results on CIFAR10

  1. Goodfellow, I. J., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. Maxout Networks. In Proceedings of the 30th International Conference on Machine Learning (ICML), pp. 1319–1327, 2013.
  2. Wan, L., Zeiler, M., Zhang, S., LeCun, Y., and Fergus, R. Regularization of neural networks using dropconnect. International Conference on Machine Learning (ICML), 2013.
  3. Lin, M., Chen, Q., and Yan, S. Network In Network. International Conference on Learning Representations (ICLR), 2014.
  4. Springenberg, J.T., Dosovitskiy, A., Brox, T., and Riedmiller, M. Striving for Simplicity: The All Convolutional Net. Proceedings of the International Conference on Learning Representations (ICLR), 2015.
  5. Clevert, D., Unterthiner, T., and Hochreiter, S. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). arXiv:1511.07289v3, 2015.
  6. Lee, C., Xie, S., Gallagher, P.W., Zhang, Z., and Tu, Z. Deeply-Supervised Nets. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics (AISTATS), 2015.
  7. Lee, C., Gallagher, P. W., and Tu, Z. Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree. ArXiv:1509.08985, 2015.
  8. Srivastava, Rupesh Kumar, Greff, Klaus, and Schmidhuber, Jurgen. Training Very Deep Networks. Advances in Neural Information Processing Systems (NIPS), 2015.
  9. Zagoruyko, S. and Komodakis, N. Wide Residual Networks. arXiv:1605.07146, 2016.
  10. He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition. arXiv:1512.03385, 2015.
  11. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Identity Mappings in Deep Residual Networks. arXiv:1603.05027, 2016.