Awesome

A Comprehensive Overhaul of Feature Distillation

Accepted at ICCV 2019

Official PyTorch implementation of "A Comprehensive Overhaul of Feature Distillation" | paper | project page | blog

Byeongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, Jin Young Choi

Clova AI Research, NAVER Corp.
Seoul National University

Requirements

• Python3
• PyTorch (> 0.4.1)
• torchvision
• numpy
• scipy

Updates

19 Nov 2019 Segmentation released

10 Sep 2019 Initial upload

CIFAR-100

Settings

We provide the code of the experimental settings specified in the paper.

Teacher models

Download following pre-trained teacher network and put them into ./data directory

• Wide Residual Network 28-4
• PyramidNet-200(240)

Training

Run CIFAR-100/train_with_distillation.py with setting alphabet (a - f)

cd CIFAR-100
python train_with_distillation.py \
--setting a \
--epochs 200 \
--batch_size 128 \
--lr 0.1 \
--momentum 0.9 \
--weight_decay 5e-4

For pyramid teacher (e, f), we used batch-size 64 to save gpu memory.

cd CIFAR-100
python train_with_distillation.py \
--setting e \
--epochs 200 \
--batch_size 64 \
--lr 0.1 \
--momentum 0.9 \
--weight_decay 5e-4

Experimental results

Performance measure is classification error rate (%)

ImageNet

Settings

In case of ImageNet, teacher model will be automatically downloaded from PyTorch sites.

Training

• (a) : ResNet152 to ResNet50

cd ImageNet
python train_with_distillation.py \
--data_path your/path/to/ImageNet \
--net_type resnet \
--epochs 100 \
--lr 0.1 \
--batch_size 256

• (b) : ResNet50 to MobileNet

cd ImageNet
python train_with_distillation.py \
--data_path your/path/to/ImageNet \
--net_type mobilenet \
--epochs 100 \
--lr 0.1 \
--batch_size 256

Experimental results

• ResNet 50

• MobileNet

Segmentation - Pascal VOC

Our segmentation code is based on pytorch-deeplab-xception.

Additional requirements

• tqdm
• matplotlib
• pillow

Settings

Teacher models

Download following pre-trained teacher network and put it into ./Segmentation/pretrained directory

• ResNet101-DeepLabV3+

We used pre-trained model in pytorch-deeplab-xception for teacher network.

Training

• First, move to segmentation folder : cd Segmentation

• Next, configure your dataset path on Segmentation/mypath.py

• Without distillation

  • ResNet 18

  CUDA_VISIBLE_DEVICES=0,1 python train.py --backbone resnet18 --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov
  

  • MobileNetV2

  CUDA_VISIBLE_DEVICES=0,1 python train.py --backbone mobilenet --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov
  

• Distillation

  • ResNet 18

  CUDA_VISIBLE_DEVICES=0,1 python train_with_distillation.py --backbone resnet18 --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov
  

  -MobileNetV2

  CUDA_VISIBLE_DEVICES=0,1 python train_with_distillation.py --backbone mobilenet --gpu-ids 0,1 --dataset pascal --use-sbd --nesterov
  

Experimental results

This numbers are based validation performance of our code.

• ResNet 18

• MobileNetV2

In the paper, we reported performance on the test set, but our code measures the performance on the val set. Therefore, the performance on code is not same as the paper. If you want accurate measure, please measure performance on test set with Pascal VOC evaluation server.

Citation

@inproceedings{heo2019overhaul,
  title={A Comprehensive Overhaul of Feature Distillation},
  author={Heo, Byeongho and Kim, Jeesoo and Yun, Sangdoo and Park, Hyojin and Kwak, Nojun and Choi, Jin Young},
  booktitle = {International Conference on Computer Vision (ICCV)},
  year={2019}
}

License

Copyright (c) 2019-present NAVER Corp.

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.