Awesome

• A considerably faster version based on CUDA can be found here - Mish CUDA (All credits to Thomas Brandon for the same)
• Memory Efficient Experimental version of Mish can be found here
• Faster variants for Mish and H-Mish by Yashas Samaga can be found here - ConvolutionBuildingBlocks
• Alternative (experimental improved) variant of H-Mish developed by Páll Haraldsson can be found here - H-Mish (Available in Julia)
• Variance based initialization method for Mish (experimental) by Federico Andres Lois can be found here - Mish_init

• [07/17] Mish added to OpenVino - Open-1187, Merged-1125
• [07/17] Mish added to BetaML.jl
• [07/17] Loss Landscape exploration progress in collaboration with Javier Ideami and Ajay Uppili Arasanipalai <br>
• [07/17] Poster accepted for presentation at DLRLSS hosted by MILA, CIFAR, Vector Institute and AMII
• [07/20] Mish added to Google's AutoML - 502
• [07/27] Mish paper accepted to 31st British Machine Vision Conference (BMVC), 2020. ArXiv version to be updated soon.
• [08/13] New updated PyTorch benchmarks and pretrained models available on PyTorch Benchmarks.
• [08/14] New updated Arxiv version of the paper is out.
• [08/18] Mish added to Sony Nnabla - Merged-700
• [09/02] Mish added to TensorFlow Swift APIs - Merged - 1068
• [06/09] Official paper and presentation video for BMVC is released at this link.
• [23/09] CSP-p7 + Mish (multi-scale) is currently the SOTA in Object Detection on MS-COCO test-dev while CSP-p7 + Mish (single-scale) is currently the 3rd best model in Object detection on MS-COCO test dev. Further details on paperswithcode leaderboards.
• [11/11] Mish added to TFLearn - Merged 1159 (Follow up 1141)
• [17/11] Mish added to MONAI - Merged 1235
• [20/11] Mish added to plaidml - Merged 1566
• [10/12] Mish added to Simd and Synet - Docs
• [14/12] Mish added to OneFlow - Merged 3972
• [24/12] Mish added to GPT-Neo
• [21/04] Mish added to TensorFlow JS
• [02/05] Mish added to Axon
• [26/05] 🔥 Mish is added to PyTorch. Will be added in PyTorch 1.9. 🔥
• [27/05] Mish is added to PyTorch YOLO v3
• [09/06] 🔥 Mish is added to MXNet.
• [03/07] Mish is added to TorchSharp.
• [05/08] Mish is added to KotlinDL.

News/ Media Coverage:

• (02/2020): Podcast episode on Mish at Machine Learning Café is out now. Listen on:

• (02/2020): Talk on Mish and Non-Linear Dynamics at Sicara is out now. Watch on:

• (07/2020): CROWN: A comparison of morphology for Mish, Swish and ReLU produced in collaboration with Javier Ideami. Watch on:

• (08/2020): Talk on Mish and Non-Linear Dynamics at Computer Vision Talks. Watch on:

• (12/2020): Talk on From Smooth Activations to Robustness to Catastrophic Forgetting at Weights & Biases Salon is out now. Watch on:

• (12/2020) Weights & Biases integration is now added 🔥. Get started.

• (08/2021) Comprehensive hardware based computation performance benchmark for Mish has been conducted by Benjamin Warner. Blogpost.

1. Mish <br> a. Loss landscape
2. ImageNet Scores
3. MS-COCO
4. Variation of Parameter Comparison<br> a. MNIST<br> b. CIFAR10<br>
5. Significance Level <br>
6. Results<br> a. Summary of Results (Vision Tasks)<br> b. Summary of Results (Language Tasks)<br>
7. Try It!<br>
8. Acknowledgements
9. Cite this work

Mish:

Minimum of f(x) is observed to be ≈-0.30884 at x≈-1.1924<br> Mish has a parametric order of continuity of: C^∞

Derivative of Mish with respect to Swish and Δ(x) preconditioning:

Further simplifying:

<a href="https://www.codecogs.com/eqnedit.php?latex=f'(x)&space;=&space;\Delta(x)swish(x)&space;&plus;&space;\frac{f(x)}{x}" target="_blank"><img src="https://latex.codecogs.com/svg.latex?f'(x)&space;=&space;\Delta(x)swish(x)&space;&plus;&space;\frac{f(x)}{x}" title="f'(x) = \Delta(x)swish(x) + \frac{f(x)}{x}" /></a>

Alternative derivative form:

<a href="https://www.codecogs.com/eqnedit.php?latex=f'(x)&space;=&space;\frac{e^{x}\omega}{\delta^{2}}" target="_blank"><img src="https://latex.codecogs.com/svg.latex?f'(x)&space;=&space;\frac{e^{x}\omega}{\delta^{2}}" title="f'(x) = \frac{e^{x}\omega}{\delta^{2}}" /></a>

where:

<a href="https://www.codecogs.com/eqnedit.php?latex=\omega&space;=&space;4(x&plus;1)&plus;4e^{2x}&space;&plus;e^{3x}&space;&plus;e^{x}(4x&plus;6)" target="_blank"><img src="https://latex.codecogs.com/svg.latex?\omega&space;=&space;4(x&plus;1)&plus;4e^{2x}&space;&plus;e^{3x}&space;&plus;e^{x}(4x&plus;6)" title="\omega = 4(x+1)+4e^{2x} +e^{3x} +e^{x}(4x+6)" /></a>

<a href="https://www.codecogs.com/eqnedit.php?latex=\delta&space;=&space;2e^{x}&space;&plus;e^{2x}&space;&plus;2" target="_blank"><img src="https://latex.codecogs.com/svg.latex?\delta&space;=&space;2e^{x}&space;&plus;e^{2x}&space;&plus;2" title="\delta = 2e^{x} +e^{2x} +2" /></a>

We hypothesize the Δ(x) to be exhibiting the properties of a pre-conditioner making the gradient more smoother. Further details are provided in the paper.

Loss Landscape:

To visit the interactive Loss Landscape visualizer, click here.

Loss landscape visualizations for a ResNet-20 for CIFAR 10 using ReLU, Mish and Swish (from L-R) for 200 epochs training:

Mish provides much better accuracy, overall lower loss, smoother and well conditioned easy-to-optimize loss landscape as compared to both Swish and ReLU. For all loss landscape visualizations please visit this readme.

We also investigate the output landscape of randomly initialized neural networks as shown below. Mish has a much smoother profile than ReLU.

ImageNet Scores:

For Installing DarkNet framework, please refer to darknet(Alexey AB)

For PyTorch based ImageNet scores, please refer to this readme

Results on CSPResNext-50:

Results on CSPResNet-50:

Results on CSPDarkNet-53:

Results on SpineNet-49:

MS-COCO:

For PyTorch based MS-COCO scores, please refer to this readme

Credits to AlexeyAB, Wong Kin-Yiu and Glenn Jocher for all the help with benchmarking MS-COCO and ImageNet.

Variation of Parameter Comparison:

MNIST:

To observe how increasing the number of layers in a network while maintaining other parameters constant affect the test accuracy, fully connected networks of varying depths on MNIST, with each layer having 500 neurons were trained. Residual Connections were not used because they enable the training of arbitrarily deep networks. BatchNorm was used to lessen the dependence on initialization along with a dropout of 25%. The network is optimized using SGD on a batch size of 128, and for fair comparison, the same learning rates for each activation function was maintained. In the experiments, all 3 activations maintained nearly the same test accuracy for 15 layered Network. Increasing number of layers from 15 gradually resulted in a sharp decrease in test accuracy for Swish and ReLU, however, Mish outperformed them both in large networks where optimization becomes difficult.

The consistency of Mish providing better test top-1 accuracy as compared to Swish and ReLU was also observed by increasing Batch Size for a ResNet v2-20 on CIFAR-10 for 50 epochs while keeping all other network parameters to be constant for fair comparison.

Gaussian Noise with varying standard deviation was added to the input in case of MNIST classification using a simple conv net to observe the trend in decreasing test top-1 accuracy for Mish and compare it to that of ReLU and Swish. Mish mostly maintained a consistent lead over that of Swish and ReLU (Less than ReLU in just 1 instance and less than Swish in 3 instance) as shown below. The trend for test loss was also observed following the same procedure. (Mish has better loss than both Swish and ReLU except in 1 instance)

CIFAR10:

Significance Level:

The P-values were computed for different activation functions in comparison to that of Mish on terms of Top-1 Testing Accuracy of a Squeeze Net Model on CIFAR-10 for 50 epochs for 23 runs using Adam Optimizer at a Learning Rate of 0.001 and Batch Size of 128. It was observed that Mish beats most of the activation functions at a high significance level in the 23 runs, specifically it beats ReLU at a high significance of P < 0.0001. Mish also had a comparatively lower standard deviation across 23 runs which proves the consistency of performance for Mish.

Values rounded up which might cause slight deviation in the statistical values reproduced from these tests

Results:

News: Ajay Arasanipalai recently submitted benchmark for CIFAR-10 training for the Stanford DAWN Benchmark using a Custom ResNet-9 + Mish which achieved 94.05% accuracy in just 10.7 seconds in 14 epochs on the HAL Computing Cluster. This is the current fastest training of CIFAR-10 in 4 GPUs and 2nd fastest training of CIFAR-10 overall in the world.

Summary of Results (Vision Tasks):

Comparison is done based on the high priority metric, for image classification the Top-1 Accuracy while for Generative Networks and Image Segmentation the Loss Metric. Therefore, for the latter, Mish > Baseline is indicative of better loss and vice versa. For Embeddings, the AUC metric is considered.

Summary of Results (Language Tasks):

Comparison is done based on the best metric score (Test accuracy) across 3 runs.

Try It!

Thanks to all the people who have helped and supported me massively through this project who include:

1. Sparsha Mishra
2. Alexandra Deis
3. Alexey Bochkovskiy
4. Chien-Yao Wang
5. Thomas Brandon
6. Less Wright
7. Manjunath Bhat
8. Ajay Uppili Arasanipalai
9. Federico Lois
10. Javier Ideami
11. Ioannis Anifantakis
12. George Christopoulos
13. Miklos Toth

And many more including the Fast AI community, Weights and Biases Community, TensorFlow Addons team, SpaCy/Thinc team, Sicara team, Udacity scholarships team to name a few. Apologies if I missed out anyone.

Cite this work:

@article{misra2019mish,
  title={Mish: A self regularized non-monotonic neural activation function},
  author={Misra, Diganta},
  journal={arXiv preprint arXiv:1908.08681},
  year={2019}
}