Awesome

🔄 RoLoRA

This repository contains the code of RoLoRA introduced in our work: "RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization", published in EMNLP 2024.

🌟 Abstract

In this work, we propose RoLoRA, the first LoRA-based scheme to apply rotation for outlier elimination, and then fine-tune rotated outlier-free LLMs for effective weight-activation quantization. RoLoRA can improve low-bit LoRA convergence and post-training quantization robustness in weight-activation quantization settings. RoLoRA is evaluated across various LLM series, tasks, and quantization settings, achieving up to 29.5% absolute accuracy gain of 4-bit weight-activation quantization of LLaMA2-13B on commonsense reasoning tasks compared to LoRA baseline.

🌿 Citation

If you find our code useful for your research, please consider citing:

@article{huang2024rolora,
  title={RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization},
  author={Huang, Xijie and Liu, Zechun and Liu, Shih-Yang and Cheng, Kwang-Ting},
  journal={arXiv preprint arXiv:2407.08044},
  year={2024}
}

🛠️ Getting Started

Huggingface Hub Login

pip install --upgrade huggingface_hub
huggingface-cli login

Installation

pip install -r requirements.txt

If you encounter any problems installing fast_hadamard_transform using pip, please consider building from source

🚂 Finetuning

For experiments applying RoLoRA on LLaMA2-7B, please run

sh rolora.sh

Remove --rotate_down_proj and --rotate_mode 'hadamard' for LoRA baseline without rotation.

⌛ Merging

To merge RoLoRA adapter to LLaMA2-7B, please run

sh merge_rolora.sh

Specify --adapter_name_or_path and --export_dir to be path of adapter files and export target folder. Remove --rotate_down_proj and --rotate_mode 'hadamard' for merging LoRA adapter without rotation.

🔍 Evaluation

For evaluation on Zero-shot CommonSense Reasoning (ZCSR) and MMLU benchmarks, please run

sh eval_rolora.sh

Specify $NAME, $WBITS, and $ABITS for the target quantization settings. Use --w_rtn for RTN quantization on weights (default is GPTQ). If you want evaluate the quantized models on more tasks, modify --task to any tasks that are included in lm-evaluation-harness.

💾 Checkpoint

We provide the checkpoints for the RoLoRA-finetuned LLMs in the given huggingface repo. The evaluation logs are also included.

• ScarletAce/LLaMA2-7B-RoLoRA
• ScarletAce/LLaMA2-13B-RoLoRA
• ScarletAce/LLaMA3-8B-RoLoRA

📚 Results

Below is the results in LLaMA2-7B, LLaMA2-13B, and LLaMA3-8B on zero-shot commonsense reasoning（ZCSR）and MMLU benchmarks.

💌 Acknowledgement

This repo benefits from SpinQuant, QuaRot, LLaMa-Factory, and fast-hadamard-transform. Thanks for their wonderful works!

If you have any questions, feel free to contact Xijie Huang (xhuangbs AT connect.ust.hk, huangxijie1108 AT gmail.com)