Awesome
EdgeQAT
Official repo for the paper: EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge
Implementation
Follow the instructions of the BabyLLaMA to implement the training environment, and BabyLM Challenge to implement the evaluation environment.
Usage
- Download dataset from BabyLM Challenge
- Clean the dataset according to BabyLLaMA
- Pretrain teacher model
- Download FP16 LLaMA-58M model from BabyLLaMA
- QAT with scripts in
distill_train/scripts/ - Evaluation with scripts in
evaluation_pipeline/
Citation
@article{shen2024edgeqat,
title={EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge},
author={Shen, Xuan and Kong, Zhenglun and Yang, Changdi and Han, Zhaoyang and Lu, Lei and Dong, Peiyan and others},
journal={arXiv preprint arXiv:2402.10787},
year={2024}
}