Home

Awesome

OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models

<h5 align="center">

arXiv jiqizhixin zhihu License GitHub Stars🔥🔥🔥 <br>

</h5>

omniquant

OmniQuant is a simple and powerful quantization technique for LLMs. The current release supports:

News

Contents

Install

conda create -n omniquant python=3.10 -y
conda activate omniquant
git clone https://github.com/OpenGVLab/OmniQuant.git
cd OmniQuant
pip install --upgrade pip 
pip install -e .

We also leverage the kernel from AutoGPTQ to achieve real quantization. So you should also install the bug-fixed AutoGPTQ as follows::

git clone https://github.com/ChenMnZ/AutoGPTQ-bugfix
pip install -v .

OmniQuant Model Zoo

We provide pre-trained Omniquant model zoo for multiple model families, including LLaMa-1&2, LLaMa-2-Chat, OPT.

You can download the pre-trained OmniQuant parameters you need at Huggingface.

The detailed support list:

ModelsSizesW2A16W2A16g128W2A16g64W3A16
LLaMA7B/13B/30B/65B✅✅✅✅
LLaMA-27B/13B/70B✅✅✅✅
OPT125m/1.3B/2.7B/6.7B/13B/30B/66B✅✅✅✅
ModelsSizesW3A16g128W4A16W4A16g128W6A6W4A4
LLaMA7B/13B/30B/65B✅✅✅✅✅
LLaMA-27B/13B/70B✅✅✅✅✅
OPT125m/1.3B/2.7B/6.7B/13B/30B/66B✅✅✅✅✅
LLaMA-2-Chat7B/13B✅

Usage

We provide full script to run OmniQuant in ./scripts/. We use LLaMa-7B as an example here:

  1. Obtain the channel-wise scales and shifts required for initialization:
conda install git git-lfs
git lfs install
git clone https://huggingface.co/ChenMnZ/act_shifts
git clone https://huggingface.co/ChenMnZ/act_scales

Optional, we also offer the script that you can generate channel-wise scales and shifts by yourself:

python generate_act_scale_shift.py --model /PATH/TO/LLaMA/llama-7b
  1. Weight-only quantization
# W3A16
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/LLaMA/llama-7b  \
--epochs 20 --output_dir ./log/llama-7b-w3a16 \
--eval_ppl --wbits 3 --abits 16 --lwc

# W3A16g128
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/LLaMA/llama-7b  \
--epochs 20 --output_dir ./log/llama-7b-w3a16g128 \
--eval_ppl --wbits 3 --abits 16 --group_size 128 --lwc
  1. weight-activation quantization
# W4A4
CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/LLaMA/llama-7b  \
--epochs 20 --output_dir ./log/llama-7b-w4a4 \
--eval_ppl --wbits 4 --abits 4 --lwc --let \
--tasks piqa,arc_easy,arc_challenge,boolq,hellaswag,winogrande
  1. reproduce evaluation results of our paper

    1) download the pretrained OmniQuant parameters you want through Huggingface.

    2) set epoch as 0 and inference with resume, take LLaMa-7B with W3A16g128 quantization as an example:

CUDA_VISIBLE_DEVICES=0 python main.py \
--model /PATH/TO/LLaMA/llama-7b  \
--epochs 0 --output_dir ./log/test \
--eval_ppl --wbits 3 --abits 16 --group_size 128 --lwc \
--resume /PATH/TO/Pretrained/Parameters 

More detailed and optional arguments:

Runing Quantized Models with MLC-LLM

MLC-LLM offers a universal deployment solution suitable for various language models across a wide range of hardware backends, encompassing iPhones, Android phones, and GPUs from NVIDIA, AMD, and Intel.

We compile the OmniQuant's quantization models through MLC-LLM and offer an out-of-the-box case here. You can see smaller gpu memory usage and inference speedup. Detailed instructions can be found in in runing_quantized_models_with_mlc_llm.ipynb.

Specially, we also deploy the aforementioned two quantized models into mobile phones through MLC-LLM. You can download the Android app by simply clicking the button below:

<img src="./imgs/download.png" width="150"/>

This app includes three models, LLaMa-2-7B-Chat-Omniquant-W3A16g128asym, LLaMa-2-13B-Chat-Omniquant-W3A16g128asym, and LLaMa-2-13B-Chat-Omniquant-W2A16g128asym. They require at least 4.5G, 7.5G, and 6.0G free RAM, respectively. Note that 2bit quantization has worse performance compared to 3bit quantization as shown in our paper. The inclusion of 2-bit quantization is just an extreme exploration about deploy LLM in mobile phones. Currently, this app is in its demo phase and may experience slower response times, so wait patiently for the generation of response. We have tested this app on Redmi Note 12 Turbo (Snapdragon 7+ Gen 2 and 16G RAM), some examples are provided below:

<div style="text-align: center;"> <img src="./imgs/7b_3bit_android.png" width="500" /> </div> <div style="text-align: center;"> <img src="./imgs/13b_3bit_android.png" width="500" /> </div> <div style="text-align: center;"> <img src="./imgs/13b_2bit_android.png" width="500" /> </div>

We also have tested this app on iPhone 14 Pro (A16 Bionic and 6G RAM), some examples are provided below:

<div style="text-align: center;"> <img src="./imgs/7b_3bit_iphone.png" width="500" /> </div>

Results

Related Project

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers

RPTQ: Reorder-Based Post-Training Quantization for Large Language Models

MLC LLM

AutoGPTQ

Citation

If you use our OmniQuant approach in your research, please cite our paper:

@article{OmniQuant,
  title={OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models},
  author={Shao, Wenqi and Chen,Mengzhao and  Zhang, Zhaoyang and Xu, Peng and Zhao, Lirui and Li, Zhiqian and Zhang, Kaipeng Zhang, and Gao, Peng, and Qiao, Yu, and Luo, Ping},
  journal={arXiv preprint arXiv:2308.13137},
  year={2023}
}