Home

Awesome

SELUs (scaled exponential linear units) - Visualized and Histogramed Comparisons among ReLU and Leaky ReLU

Descriptions

This project includes a Tensorflow implementation of SELUs (scaled exponential linear units) proposed in this paper Self-Normalizing Neural Networks. Also, aiming to present clear at a glance comparisons among SELU, ReLU, Leaky ReLU, etc, this implementation focuses on visualizing and histogramming activations on Tensorboard. As a result, the drawn visualization and histogram are nicely incorporating with Tensorboard by introducing plotting summaries. Examples of visualization and histogram are as follows.

<img src="figure/AVH.png" height="450"/>,

Ideally, desire activations of every layer are close to zero mean and unit variance to make tensors propagated through several layers converge towards zero mean and unit variance. The learning can, therefore, be stabilized by preventing gradients from being vanishing and exploding. In this work, the authors propose scaled exponential linear units (SELUs) which aim to automatically shift and rescale neuron activations towards zero mean and unit variance without explicit normalization like what batch normalization technique does.

Intending to empirically verify the effectiveness of the proposed activations, a convolutional neural network consisting of three convolutional layers followed by three fully connected layers was implemented to be trained on image classification tasks on datasets such as MNIST, SVHN, and CIFAR10. To overcome the limited content allowed to be shown on Tensorboard, a plotting library Tensorflow Plot aiming to bridge the gap between Python plotting libraries and Tensorboard is introduced. Again, here are some examples.

<img src="figure/H.png" width="300"/>,

<img src="figure/V.png" width="300"/>,

The implemented model is trained and tested on three publicly available datasets: MNIST, SVHN, and CIFAR-10.

*This code is still being developed and subject to change.

Prerequisites

Usage

Datasets

Download datasets with:

$ python download.py --dataset MNIST SVHN CIFAR10

Just do it

Simply run comparisons among the default activations including SELU, ReLU, and Leaky ReLU

python script.py

Note that this script will

Use your own settings

You can change several setting with the args including batch size, learning rate and the weight decay applied to it, dataset, activations, etc. Also, if you want to test other model architectures or other activations such as sigmoid or tanh, it's also easy.

Here are some examples:

Train models with different activation functions with downloaded datasets:

$ python trainer.py --dataset MNIST --activation relu --learning_rate 1e-3
$ python trainer.py --dataset SVHN --activation lrelu --batch_size 128
$ python trainer.py --dataset CIFAR10 --activation selu --lr_weight_decay

Train and test your own datasets:

$ mkdir datasets/YOUR_DATASET
$ python trainer.py --dataset YOUR_DATASET
$ python evaler.py --dataset YOUR_DATASET

Results

Only the histogram and visualized activations of the last convolutional layer (3rd layer) and the first fully connected layer (4th layer) are selected to be presented here. (Trained for 10k iterations)

SELU

<img src="figure/result/selu3.png" width="650"/> <img src="figure/result/selu4.png" width="650"/>

ReLU

<img src="figure/result/relu3.png" width="650"/> <img src="figure/result/relu4.png" width="650"/>

Leaky ReLU

<img src="figure/result/relu3.png" width="650"/> <img src="figure/result/relu4.png" width="650"/>

Related works

Author

Shao-Hua Sun / @shaohua0116 @ Joseph Lim's research lab @ USC

Acknowledgement

The code monitor.py was written by @wookayin