Awesome

Personalized Education: Blind Knowledge Distillation

This repository is a PyTorch implementation of "Personalized Education: Blind Knowledge Distillation". The pretrained teachers are downloaded from SSKD. You can also download from pretrained teachers.

Requirements

The code was tested on

Python 3.6
torch 1.2.0
torchvision 0.4.0

Training

To train small students from scratch by distilling knowledge from teacher networks with PE, first download the pretrained teachers into the "pretrained_teachers" folder, and then simply run the command below:<br>sh train.sh

Hyperparameters on other datasets

Tiny ImageNet: --weight_decay 0.0001 --pro 0.3 --alphas '0.1, 0.5, 1.0' --ps '64, 32' --search_T 4 --kd_T 4 --epochs 100 --lr_decay_epochs '30,60,90' --updata_epoch 30<br> ImageNet: --weight_decay 0.00005 --pro 0.5 --alphas '0.1, 0.3' --ps '224, 112' --epochs 100 --search_T 2 --kd_T 2 --lr_decay_epochs '30,60,90' --updata_epoch 20

Notes

PE-BKD is a general framework. You can design your own prior region based on the prior knowledge that you have about your target task.

Citation

If you find this code helpful, you may consider citing this paper:

@inproceedings{deng2022personalized,
  title={Personalized Education: Blind Knowledge Distillation},
  author={Deng, Xiang and Zheng, Jian and Zhang, Zhongfei},
  booktitle = {Proceedings of the 2022 European Conference on Computer Vision},
  year={2022}
}