Awesome

Introduction

KAmalEngine (KAE) aims at building a lightweight algorithm package for Knowledge Amalgamation, Knowledge Distillation and Model Transferability Estimation.

Features:

• Algorithms for knowledge amalgamation and distillation
• Deep model transferability estimation based on attribution maps
• Predefined callbacks & metrics for evaluation and visualization
• Easy-to-use tools for multi-tasking training, e.g. synchronized transformation

Table of contents

• Introduction
• Table of contents
• Quick Start
• Algorithms
  • 1. Knowledge Amalgamation
  • 2. Knowledge Distillation
  • 3. Model Transferability
• Transferability Graph
• Authors

Quick Start

Please follow the instructions in QuickStart.md for the basic usage of KAE. More examples can be found in examples, including knowledge amalgamation, knowledge distillation, transferability and model slimming.

Algorithms

1. Knowledge Amalgamation

Amalgamating Knowledge towards Comprehensive Classification (AAAI 2019)

@inproceedings{shen2019amalgamating,
  author={Shen, Chengchao and Wang, Xinchao and Song, Jie and Sun, Li and Song, Mingli},
  title={Amalgamating Knowledge towards Comprehensive Classification},
  booktitle={AAAI Conference on Artificial Intelligence (AAAI)},
  pages={3068--3075},
  year={2019}
}

Scripts can be found in examples/amalgamation/layerwise_ka

Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation (ICCV 2019)

@inproceedings{shen2019customizing,
  title={Customizing student networks from heterogeneous teachers via adaptive knowledge amalgamation},
  author={Shen, Chengchao and Xue, Mengqi and Wang, Xinchao and Song, Jie and Sun, Li and Song, Mingli},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={3504--3513},
  year={2019}
}

Scripts can be found in examples/amalgamation/customize_ka

Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More (CVPR 2019)

@inproceedings{ye2019student,
  title={Student Becoming the Master: Knowledge Amalgamation for Joint Scene Parsing, Depth Estimation, and More},
  author={Ye, Jingwen and Ji, Yixin and Wang, Xinchao and Ou, Kairi and Tao, Dapeng and Song, Mingli},
  booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition},
  pages={2829--2838},
  year={2019}
}

Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning (IJCAI 2019)

Feature Space	Common Space

@inproceedings{luo2019knowledge,
  title={Knowledge Amalgamation from Heterogeneous Networks by Common Feature Learning},
  author={Luo, Sihui and Wang, Xinchao and Fang, Gongfan and Hu, Yao and Tao, Dapeng and Song, Mingli},
  booktitle={Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI)},
  year={2019},
}

Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning (ECCV 2020)

@inproceedings{luo2020collaboration,
  title={Collaboration by competition: Self-coordinated knowledge amalgamation for multi-talent student learning},
  author={Luo, Sihui and Pan, Wenwen and Wang, Xinchao and Wang, Dazhou and Tang, Haihong and Song, Mingli},
  booktitle={Computer Vision--ECCV 2020: 16th European Conference, Glasgow, UK, August 23--28, 2020, Proceedings, Part VI 16},
  pages={631--646},
  year={2020},
  organization={Springer}
}

Scripts can be found in examples/knowledge_amalgamation/soka

Knowledge Amalgamation for Object Detection With Transformers (TIP 2021)

@article{zhang2023knowledge,
  title={Knowledge Amalgamation for Object Detection With Transformers},
  author={Zhang, Haofei and Mao, Feng and Xue, Mengqi and Fang, Gongfan and Feng, Zunlei and Song, Jie and Song, Mingli},
  journal={IEEE Transactions on Image Processing},
  volume={32},
  pages={2093--2106},
  year={2023},
  publisher={IEEE}
}

Scripts can be found in examples/amalgamation/transformer_ka

Federated Selective Aggregation for Knowledge Amalgamation (CHIP 2022)

@article{XIE2023100053,
  title = {Federated selective aggregation for on-device knowledge amalgamation},
  journal = {Chip},
  volume = {2},
  number = {3},
  pages = {100053},
  year = {2023},
  issn = {2709-4723},
  doi = {https://doi.org/10.1016/j.chip.2023.100053},
  url = {https://www.sciencedirect.com/science/article/pii/S2709472323000163},
  author = {Donglin Xie and Ruonan Yu and Gongfan Fang and Jiaqi Han and Jie Song and Zunlei Feng and Li Sun and Mingli Song}
}

2. Knowledge Distillation

Hearing Lips: Improving Lip Reading by Distilling Speech Recognizers (AAAI 2020)

@inproceedings{zhao2020hearing,
  title={Hearing lips: Improving lip reading by distilling speech recognizers},
  author={Zhao, Ya and Xu, Rui and Wang, Xinchao and Hou, Peng and Tang, Haihong and Song, Mingli},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={34},
  number={04},
  pages={6917--6924},
  year={2020}
}

Scripts can be found in examples/distillation/libs

Progressive Network Grafting for Few-Shot Knowledge Distillation (AAAI 2021)

@inproceedings{shen2021progressive,
  title={Progressive network grafting for few-shot knowledge distillation},
  author={Shen, Chengchao and Wang, Xinchao and Yin, Youtan and Song, Jie and Luo, Sihui and Song, Mingli},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={35},
  number={3},
  pages={2541--2549},
  year={2021}
}

Scripts can be found in /examples/distillation/graft_kd

KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation (IJCAI 2021)

@inproceedings{ijcai2021p444,
  title     = {KDExplainer: A Task-oriented Attention Model for Explaining Knowledge Distillation},
  author    = {Xue, Mengqi and Song, Jie and Wang, Xinchao and Chen, Ying and Wang, Xingen and Song, Mingli},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Zhi-Hua Zhou},
  pages     = {3228--3234},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/444},
  url       = {https://doi.org/10.24963/ijcai.2021/444},
}

Scripts can be found in /examples/distillation/kd_explainer

Contrastive Model Inversion for Data-Free Knowledge Distillation (IJCAI 2021)

@inproceedings{ijcai2021p327,
  title     = {Contrastive Model Invertion for Data-Free Knolwedge Distillation},
  author    = {Fang, Gongfan and Song, Jie and Wang, Xinchao and Shen, Chengchao and Wang, Xingen and Song, Mingli},
  booktitle = {Proceedings of the Thirtieth International Joint Conference on
               Artificial Intelligence, {IJCAI-21}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Zhi-Hua Zhou},
  pages     = {2374--2380},
  year      = {2021},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2021/327},
  url       = {https://doi.org/10.24963/ijcai.2021/327},
}

Scripts can be found in examples/distillation/cmi

Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data (NeurIPS 2021)

@article{fang2021mosaicking,
  title={Mosaicking to distill: Knowledge distillation from out-of-domain data},
  author={Fang, Gongfan and Bao, Yifan and Song, Jie and Wang, Xinchao and Xie, Donglin and Shen, Chengchao and Song, Mingli},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  pages={11920--11932},
  year={2021}
}

Up to 100× Faster Data-free Knowledge Distillation (AAAI 2022)

@inproceedings{fang2022up,
  title={Up to 100x faster data-free knowledge distillation},
  author={Fang, Gongfan and Mo, Kanya and Wang, Xinchao and Song, Jie and Bei, Shitao and Zhang, Haofei and Song, Mingli},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={36},
  number={6},
  pages={6597--6604},
  year={2022}
}

Data-Free Adversarial Distillation

@article{fang2019data,
  title={Data-free adversarial distillation},
  author={Fang, Gongfan and Song, Jie and Shen, Chengchao and Wang, Xinchao and Chen, Da and Song, Mingli},
  journal={arXiv preprint arXiv:1912.11006},
  year={2019}
}

Scripts can be found in examples/distillation/dfkd

Safe Distillation Box (AAAI 2022)

@inproceedings{ye2022safe,
  title={Safe distillation box},
  author={Ye, Jingwen and Mao, Yining and Song, Jie and Wang, Xinchao and Jin, Cheng and Song, Mingli},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={36},
  number={3},
  pages={3117--3124},
  year={2022}
}

Scripts can be found in examples/distillation/sdb

Spot-adaptive Knowledge Distillation (TIP 2022)

@article{song2022spot,
  title={Spot-adaptive knowledge distillation},
  author={Song, Jie and Chen, Ying and Ye, Jingwen and Song, Mingli},
  journal={IEEE Transactions on Image Processing},
  volume={31},
  pages={3359--3370},
  year={2022},
  publisher={IEEE}
}

Scripts can be found in examples/distillation/sakd

Tree-like Decision Distillation (CVPR 2021)

@inproceedings{song2021tree,
  title={Tree-like decision distillation},
  author={Song, Jie and Zhang, Haofei and Wang, Xinchao and Xue, Mengqi and Chen, Ying and Sun, Li and Tao, Dacheng and Song, Mingli},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13488--13497},
  year={2021}
}

Script can be found in examples/distillation/tdd

Context Correlation Distillation for Lip Reading (计算机辅助设计与图形学学报)

@articleInfo{19723,
title = "针对唇语识别的上下文相关性蒸馏方法",
journal = "计算机辅助设计与图形学学报",
volume = "34",
number = "19723,
pages = "1559",
year = "2022",
note = "",
issn = "1003-9775",
doi = "10.3724/SP.J.1089.2022.19723",
url = "https://www.jcad.cn/article/doi/10.3724/SP.J.1089.2022.19723",
author = "赵雅","冯尊磊","王慧琼","宋明黎",keywords = "唇语识别","知识蒸馏","跨模态",
}

Scripts can be found in examples/distillation/c2kd

3. Model Transferability

Deep model transferability from attribution maps (NeurIPS 2019)

@inproceedings{song2019deep,
  title={Deep model transferability from attribution maps},
  author={Song, Jie and Chen, Yixin and Wang, Xinchao and Shen, Chengchao and Song, Mingli},
  booktitle={Advances in Neural Information Processing Systems},
  pages={6182--6192},
  year={2019}
}

DEPARA: Deep Attribution Graph for Deep Knowledge Transferability (CVPR 2020)

@inproceedings{song2020depara,
  title={DEPARA: Deep Attribution Graph for Deep Knowledge Transferability},
  author={Song, Jie and Chen, Yixin and Ye, Jingwen and Wang, Xinchao and Shen, Chengchao and Mao, Feng and Song, Mingli},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3922--3930},
  year={2020}
}

Transferability Graph

This is an example for deep model transferability on 300 classification models. see examples/transferability for more details.

Authors

This project is developed by VIPA Lab from Zhejiang University and Zhejiang Lab