Awesome

neuron-selectivity-transfer

This code is a re-implementation of the imagenet classification experiments in the paper Like What You Like: Knowledge Distill via Neuron Selectivity Transfer .

Citation

If you use our code in your research or wish to refer to the baseline results, please use the following BibTeX entry.

@article{NST2017
  author =   {Zehao Huang and Naiyan Wang},
  title =    {Like What You Like: Knowledge Distill via Neuron Selectivity Transfer},
  journal =  {arXiv preprint arXiv:1707.01219},
  year =     {2017}
}

Implementation

This code is implemented by a modified MXNet which supports ResNeXt-like augmentation. (This version of MXNet does not support cudnn7)

ImageNet data preparation

Download the ImageNet dataset and create pass through rec (following tornadomeet's repository but using unchange mode)

Run

bash init.sh
modify config/cfgs.py
python train.py

Results on ImageNet-1k

Single Model, Single Crop Validation Error:

Method	Model	Top-1	Top-5	Download
Student	Inception-BN	25.74	8.07	Dropbox
NST (Poly kernel)	Inception-BN	24.81	7.55	Dropbox
NST (Poly kernel) + KD	Inception-BN	24.34	7.11	Dropbox
NST (Poly kernel) + KD	A modified ResNet-50	21.05	5.56	Dropbox

Note: The symbol of our modified ResNet-50 (following SENet Appendix B) is available in Dropbox. The mean RGB for our modified ResNet-50 is [123.68, 116.28, 103.53], std is [58.395, 57.12, 57.375]. For Inception-BN, we don't need to do data pre-processing since we add a BN layer in the beginning of the network.