Home

Awesome

AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer

This repository contains the official PyTorch implementation for the ECCV2024 paper "AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer". AdaLog adapts the logarithmic base to accommodate the power-law-like distribution of activations, and simultaneously allows for hardware-friendly quantization and de-quantization. By further employing the bias reparameterization, AdaLog Quantizer is applicable to both the post-Softmax and post-GeLU activations. The Fast Progressive Combining Search (FPCS) strategy is used to select the best logarithm base for AdaLog quantizers, as well as those scaling factors and zero points for uniform quantizers.

adalog

Getting Started

git clone https://github.com/GoatWu/AdaLog.git
cd AdaLog
pip install torch==1.10.0 torchvision --index-url https://download.pytorch.org/whl/cu113
pip install timm==0.9.2

All the pretrained models can be obtained using timm. You can also directly download the checkpoints we provide. For example:

wget https://github.com/GoatWu/AdaLog/releases/download/v1.0/deit_tiny_patch16_224.bin
mkdir -p ./checkpoints/vit_raw/
mv deit_tiny_patch16_224.bin ./checkpoints/vit_raw/

Evaluation

You can quantize and evaluate a single model using the following command:

python test_quant.py --model <MODEL> --config <CONFIG_FILE> --dataset <DATA_DIR> [--calibrate] [--load-calibrate-checkpoint <CALIB_CKPT>] [--optimize]

Example: Run calibration.

python test_quant.py --model deit_tiny --config ./configs/3bit.py --dataset ~/data/ILSVRC/Data/CLS-LOC --val-batch-size 500 --calibrate

Example: Run calibration and optimization.

python test_quant.py --model deit_tiny --config ./configs/3bit.py --dataset ~/data/ILSVRC/Data/CLS-LOC --val-batch-size 500 --calibrate --optimize

Example: Load a calibrated checkpoint, and then run optimization.

python test_quant.py --model deit_tiny --config ./configs/3bit.py --dataset ~/data/ILSVRC/Data/CLS-LOC --val-batch-size 500 --load-calibrate-checkpoint ./checkpoints/quant_result/deit_tiny_w3_a3_s3_calibsize_32.pth --optimize

Example: Test an optimized checkpoint.

python test_quant.py --model deit_tiny --config ./configs/3bit.py --dataset ~/data/ILSVRC/Data/CLS-LOC --val-batch-size 500 --load-optimize-checkpoint ./checkpoints/quant_result/deit_tiny_w3_a3_s3_optimsize_1024.pth --test-optimize-checkpoint

Results

The fake-quantized checkpoint in FP32 can be obtained in Google Drive.

Quant  Model Full Prec.W3A3 (Calibrate)W3A3 (Retrain)W4A4 (Calibrate)W4A4 (Retrain)W6A6 (Calibrate)
ViT-S81.3913.8862.5072.7577.2580.91
ViT-B84.5437.9176.7379.6883.1384.80
DeiT-T72.2131.5648.7863.5267.1471.38
DeiT-S79.8524.4763.2472.0676.3679.39
DeiT-B81.8057.4574.9978.0380.2081.55
Swin-S83.2364.4176.0180.7781.9983.19
Swin-B85.2769.7578.8282.4783.8285.09

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{wu2024adalog,
title={AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer},
author={Wu, Zhuguanyu and Chen, Jiaxin and Zhong, Hanwen and Huang, Di and Wang, Yunhong},
booktitle={European Conference on Computer Vision (ECCV)},
year={2024}
}