Awesome

LSQ and LSQ+

LSQ+ net or LSQplus net and LSQ net

commit log

2023-01-08 Dorefa and Pact, https://github.com/ZouJiu1/Dorefa_Pact -------------------------------------------------------------------------------------------------------------- add torch.nn.Parameter .data, retrain models 18-01-2022

I'm not the author, I just complish an unofficial implementation of LSQ+ or LSQplus and LSQ，the origin paper you can find LSQ+ here arxiv.org/abs/2004.09576 and LSQ here arxiv.org/abs/1902.08153.

pytorch==1.8.1

You should train 32-bit float model firstly, then you can finetune a low bit-width quantization QAT model by loading the trained 32-bit float model

Dataset used for training is CIFAR10 and model used is Resnet18 revised

Version introduction

lsqplus_quantize_V1.py: initialize s、beta of activation quantization according to LSQ+ LSQ+: Improving low-bit quantization through learnable offsets and better initialization lsqplus_quantize_V2.py: initialize s、beta of activation quantization according to min max values lsqquantize_V1.py：initialize s of activation quantization according to LSQ Learned Step Size Quantization lsqquantize_V2.py: initialize s of activation quantization = 1 lsqplus_quantize_V2.py has the best result when use cifar10 dataset

The Train Results

For the below table all set a_bit=8, w_bit=8

version	weight per_channel	learning rate	A s initial	A beta initial	best epoch	Accuracy	models
Float 32bit	-	<=66 0.1<br><=86 0.01<br><=99 0.001<br><=112 0.0001	-	-	112	92.6	https://www.aliyundrive.com/s/6B2AZ45fFjx
lsqplus_quantize_V1	×	<=31 0.1<br><=61 0.01<br><=81 0.001<br><112 0.0001	1	-1e-9	90	90.3	https://www.aliyundrive.com/s/FNZRhoTe8uW
lsqplus_quantize_V2	×	as before	-	-	87	92.8	https://www.aliyundrive.com/s/WDH3ZnEa7vy
lsqplus_quantize_V1	✔	as before	-	-	96	91.19	https://www.aliyundrive.com/s/JATsi4vdurp
lsqplus_quantize_V2	✔	as before	-	-	69	92.8	https://www.aliyundrive.com/s/LRWHaBLQGWc
lsqquantize_V1	×	as before	-	-	102	91.89	https://www.aliyundrive.com/s/nR1KZZRuB23
lsqquantize_V2	×	as before	-	-	69	91.82	https://www.aliyundrive.com/s/7fjmViqUvh4
lsqquantize_V1	✔	as before	-	-	108	91.29	https://www.aliyundrive.com/s/
lsqquantize_V2	✔	as before	-	-	72	91.72	https://www.aliyundrive.com/s/7nGvMVZcKp7

all

https://www.aliyundrive.com/s/hng9XsvhYru

A represent activation, I use moving average method to initialize s and beta.

LEARNED STEP SIZE QUANTIZATION LSQ+: Improving low-bit quantization through learnable offsets and better initialization

References

https://github.com/666DZY666/micronet https://github.com/hustzxd/LSQuantization https://github.com/zhutmost/lsq-net https://github.com/Zhen-Dong/HAWQ https://github.com/KwangHoonAn/PACT https://github.com/Jermmy/pytorch-quantization-demo

Awesome

LSQ and LSQ+<br>

commit log<br>

Version introduction

The Train Results

For the below table all set a_bit=8, w_bit=8

References<br>