Awesome

<div align="center"> <h1>Awesome LLM Compression</h1> <a href="https://awesome.re"><img src="https://awesome.re/badge.svg"/></a> <img src=https://img.shields.io/github/stars/HuangOwen/Awesome-LLM-Compression.svg?style=social > <img src=https://img.shields.io/github/watchers/HuangOwen/Awesome-LLM-Compression.svg?style=social > </div>

Awesome LLM compression research papers and tools to accelerate LLM training and inference.

📑 Papers
🔧 Tools
🙌 Contributing
🌟 Star History

Papers

Survey

A Survey on Model Compression for Large Language Models TACL [Paper]
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models EMNLP 2023 [Paper] [Code]
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey Arxiv 2023 [Paper]
Efficient Large Language Models: A Survey TMLR [Paper] [GitHub Page]
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems ICML 2024 Tutorial [Paper] [Tutorial]
Understanding LLMs: A Comprehensive Overview from Training to Inference Arxiv 2024 [Paper]
Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward IJCAI 2024 (Survey Track) [Paper] [GitHub Page]
A Survey of Resource-efficient LLM and Multimodal Foundation Models Arxiv 2024 [Paper]
A Survey on Hardware Accelerators for Large Language Models Arxiv 2024 [Paper]
A Comprehensive Survey of Compression Algorithms for Language Models Arxiv 2024 [Paper]
A Survey on Transformer Compression Arxiv 2024 [Paper]
Model Compression and Efficient Inference for Large Language Models: A Survey Arxiv 2024 [Paper]
LLM Inference Unveiled: Survey and Roofline Model Insights Arxiv 2024 [Paper]
A Survey on Knowledge Distillation of Large Language Models Arxiv 2024 [Paper] [GitHub Page]
Efficient Prompting Methods for Large Language Models: A Survey Arxiv 2024 [Paper]
Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application Arxiv 2024 [Paper]
On-Device Language Models: A Comprehensive Review Arxiv 2024 [Paper] [GitHub Page] [Download On-device LLMs]
A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms Arxiv 2024 [Paper]
Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey Arxiv 2024 [Paper]
Prompt Compression for Large Language Models: A Survey Arxiv 2024 [Paper]

Quantization

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers NeurIPS 2022 [Paper] [Code (DeepSpeed)]
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale NeurIPS 2022 [Paper] [Code]
Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models NeurIPS 2022 [Paper] [Code]
LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models Arxiv 2022 [Paper]
SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models ICML 2023 [Paper] [Code]
FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantization ICML 2023 [Paper] [Code (DeepSpeed)]
Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases ICML 2023 [Paper] [Code]
The case for 4-bit precision: k-bit Inference Scaling Laws ICML 2023 [Paper]
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers ICLR 2023 [Paper] [Code]
PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models ACL 2023 [Paper]
Boost Transformer-based Language Models with GPU-Friendly Sparsity and Quantization ACL 2023 [Paper]
QLoRA: Efficient Finetuning of Quantized LLMs NeurIPS 2023 [Paper] [Code]
The Quantization Model of Neural Scaling NeurIPS 2023 [Paper]
Quantized Distributed Training of Large Models with Convergence Guarantees ICML 2023 [Paper]
RPTQ: Reorder-based Post-training Quantization for Large Language Models Arxiv 2023 [Paper] [Code]
ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation AAAI 2024 [Paper] [Code]
Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models Arxiv 2023 [Paper]
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization NeurIPS 2023 [Paper]
Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt Arxiv 2023 [Paper]
AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration MLSys 2024 (Best Paper 🏆) [Paper] [Code]
LLM-QAT: Data-Free Quantization Aware Training for Large Language Models ACL Findings 2024 [Paper] [Code]
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression ICLR 2024 [Paper] [Code]
OWQ: Lessons learned from activation outliers for weight quantization in large language models AAAI 2024 [Paper]
SqueezeLLM: Dense-and-Sparse Quantization ICML 2024 [Paper] [Code]
INT2.1: Towards Fine-Tunable Quantized Large Language Models with Error Correction through Low-Rank Adaptation Arxiv 2023 [Paper]
LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning ICLR 2024 [Paper]
INT-FP-QSim: Mixed Precision and Formats For Large Language Models and Vision Transformers Arxiv 2023 [Paper] [Code]
QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models Arxiv 2023 [Paper] [Code]
Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study COLING 2024 [Paper]
ZeroQuant-FP: A Leap Forward in LLMs Post-Training W4A8 Quantization Using Floating-Point Formats Arxiv 2023 [Paper] [Code (DeepSpeed)]
OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization ISCA 2023 [Paper]
NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search Arxiv 2023 [Paper]
GPT-Zip: Deep Compression of Finetuned Large Language Models ICML 2023 Workshop ES-FoMO [Paper]
Generating Efficient Kernels for Quantized Inference on Large Language Models ICML 2023 Workshop ES-FoMO [Paper]
Gradient-Based Post-Training Quantization: Challenging the Status Quo Arxiv 2023 [Paper]
FineQuant: Unlocking Efficiency with Fine-Grained Weight-Only Quantization for LLMs Arxiv 2023 [Paper]
OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models ICLR 2024 [Paper] [Code]
FPTQ: Fine-grained Post-Training Quantization for Large Language Models Arxiv 2023 [Paper]
eDKM: An Efficient and Accurate Train-time Weight Clustering for Large Language Models IEEE Computer Architecture Letters 2023 [Paper]
QuantEase: Optimization-based Quantization for Language Models -- An Efficient and Intuitive Algorithm Arxiv 2023 [Paper]
Norm Tweaking: High-performance Low-bit Quantization of Large Language Models AAAI 2024 [Paper]
Understanding the Impact of Post-Training Quantization on Large-scale Language Models Arxiv 2023 [Paper]
MEMORY-VQ: Compression for Tractable Internet-Scale Memory NAACL 2024 [Paper]
Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs EMNLP Findings 2024 [Paper] [Code]
Efficient Post-training Quantization with FP8 Formats MLSys 2024 [Paper] [Code (Intel® Neural Compressor)]
QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models ICLR 2024 [Paper] [Code]
Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models ICLR 2024 [Paper] [Code]
ModuLoRA: Finetuning 3-Bit LLMs on Consumer GPUs by Integrating with Modular Quantizers TMLR (Featured Certification 🌟) [Paper]
PB-LLM: Partially Binarized Large Language Models ICLR 2024 [Paper] [Code]
Dual Grained Quantization: Efficient Fine-Grained Quantization for LLM Arxiv 2023 [Paper]
QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models ICLR 2024 [Paper] [Code]
LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Models ICLR 2024 [Paper] [Code]
QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources Arxiv 2023 [Paper]
TEQ: Trainable Equivalent Transformation for Quantization of LLMs Arxiv 2023 [Paper] [Code (Intel® Neural Compressor)]
BitNet: Scaling 1-bit Transformers for Large Language Models Arxiv 2023 [Paper] [Code]
FP8-LM: Training FP8 Large Language Models Arxiv 2023 [Paper] [Code]
QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models EMNLP 2024 [Paper] [Code]
AFPQ: Asymmetric Floating Point Quantization for LLMs ACL Findings 2024 [Paper] [Code]
AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models Arxiv 2023 [Paper]
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving MLSys 2024 [Paper] [Code]
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models Arxiv 2023 [Paper]
Dissecting the Runtime Performance of the Training, Fine-tuning, and Inference of Large Language Models Arxiv 2023 [Paper]
How Does Calibration Data Affect the Post-training Pruning and Quantization of Large Language Models? Arxiv 2023 [Paper]
A Speed Odyssey for Deployable Quantization of LLMs Arxiv 2023 [Paper]
Enabling Fast 2-bit LLM on GPUs: Memory Alignment, Sparse Outlier, and Asynchronous Dequantization Arxiv 2023 [Paper]
Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing NeurIPS 2023 [Paper] [Code]
Efficient LLM Inference on CPUs NeurIPS 2023 on Efficient Natural Language and Speech Processing [Paper] [Code]
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models EMNLP Findings 2023 [Paper]
Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models EMNLP 2023 [Paper]
Revisiting Block-based Quantisation: What is Important for Sub-8-bit LLM Inference? EMNLP 2023 [Paper] [Code]
Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling EMNLP 2023 [Paper]
Watermarking LLMs with Weight Quantization EMNLP 2023 [Paper] [Code]
Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization EMNLP 2023 [Paper]
LLM-FP4: 4-Bit Floating-Point Quantized Transformers EMNLP 2023 [Paper] [Code]
Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge AAAI 2024 [Paper]
SmoothQuant+: Accurate and Efficient 4-bit Post-Training WeightQuantization for LLM Arxiv 2023 [Paper]
CBQ: Cross-Block Quantization for Large Language Models Arxiv 2023 [Paper]
ZeroQuant(4+2): Redefining LLMs Quantization with a New FP6-Centric Strategy for Diverse Generative Tasks Arxiv 2023 [Paper]
QuIP: 2-Bit Quantization of Large Language Models With Guarantees NeurIPS 2023 [Paper] [Code]
A Performance Evaluation of a Quantized Large Language Model on Various Smartphones Arxiv 2023 [Paper]
DeltaZip: Multi-Tenant Language Model Serving via Delta Compression Arxiv 2023 [Paper] [Code]
FlightLLM: Efficient Large Language Model Inference with a Complete Mapping Flow on FPGA FPGA 2024 [Paper]
Extreme Compression of Large Language Models via Additive Quantization ICML 2024 [Paper]
Quantized Side Tuning: Fast and Memory-Efficient Tuning of Quantized Large Language Models Arxiv 2024 [Paper]
Inferflow: an Efficient and Highly Configurable Inference Engine for Large Language Models Arxiv 2024 [Paper]
FP6-LLM: Efficiently Serving Large Language Models Through FP6-Centric Algorithm-System Co-Design USENIX ATC 2024 [Paper]
Can Large Language Models Understand Context? Arxiv 2024 [Paper]
EdgeQAT: Entropy and Distribution Guided Quantization-Aware Training for the Acceleration of Lightweight LLMs on the Edge Arxiv 2024 [Paper] [Code]
Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs Arxiv 2024 [Paper]
LQER: Low-Rank Quantization Error Reconstruction for LLMs ICML 2024 [Paper]
BiLLM: Pushing the Limit of Post-Training Quantization for LLMs Arxiv 2024 [Paper] [Code]
QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks ICML 2024 [Paper] [Code]
L4Q: Parameter Efficient Quantization-Aware Training on Large Language Models via LoRA-wise LSQ Arxiv 2024 [Paper]
TP-Aware Dequantization Arxiv 2024 [Paper]
ApiQ: Finetuning of 2-Bit Quantized Large Language Model EMNLP 2024 [Paper]
Accurate LoRA-Finetuning Quantization of LLMs via Information Retention Arxiv 2024 [Paper] [Code]
BitDelta: Your Fine-Tune May Only Be Worth One Bit Arxiv 2024 [Paper] [Code]
QDyLoRA: Quantized Dynamic Low-Rank Adaptation for Efficient Large Language Model Tuning EMNLP 2024 Industry Track [Paper]
Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMs ICML 2024 [Paper]
BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillation ACL 2024 [Paper] [Code]
OneBit: Towards Extremely Low-bit Large Language Models Arxiv 2024 [Paper]
DB-LLM: Accurate Dual-Binarization for Efficient LLMs Arxiv 2024 [Paper]
WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More Arxiv 2024 [Paper]
GPTVQ: The Blessing of Dimensionality for LLM Quantization Arxiv 2024 [Paper] [Code]
APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models DAC 2024 [Paper]
A Comprehensive Evaluation of Quantization Strategies for Large Language Models DAC 2024 [Paper]
Evaluating Quantized Large Language Models Arxiv 2024 [Paper]
FlattenQuant: Breaking Through the Inference Compute-bound for Large Language Models with Per-tensor Quantization Arxiv 2024 [Paper]
LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization Arxiv 2024 [Paper]
IntactKV: Improving Large Languagze Model Quantization by Keeping Pivot Tokens Intact ACL Findings 2024 [Paper] [Code]
On the Compressibility of Quantized Large Language Models Arxiv 2024 [Paper]
EasyQuant: An Efficient Data-free Quantization Algorithm for LLMs Arxiv 2024 [Paper]
What Makes Quantization for Large Language Models Hard? An Empirical Study from the Lens of Perturbation Arxiv 2024 [Paper]
SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression Arxiv 2024 [Paper] [Code]
AffineQuant: Affine Transformation Quantization for Large Language Models ICLR 2024 [Paper] [Code]
Oh! We Freeze: Improving Quantized Knowledge Distillation via Signal Propagation Analysis for Large Language Models ICLR Practical ML for Low Resource Settings Workshop 2024 [Paper]
Accurate Block Quantization in LLMs with Outliers Arxiv 2024 [Paper]
QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs Arxiv 2024 [Paper] [Code]
Minimize Quantization Output Error with Bias Compensation Arxiv 2024 [Paper] [Code]
Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models Arxiv 2024 [Paper]
Increased LLM Vulnerabilities from Fine-tuning and Quantization Arxiv 2024 [Paper]
Quantization of Large Language Models with an Overdetermined Basis Arxiv 2024 [Paper]
How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study Arxiv 2024 [Paper] [Code] [Model]
How to Parameterize Asymmetric Quantization Ranges for Quantization-Aware Training Arxiv 2024 [Paper]
Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization Arxiv 2024 [Paper] [Code]
When Quantization Affects Confidence of Large Language Models? NAACL 2024 [Paper]
QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving Arxiv 2024 [Paper] [Code]
Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs ICML 2024 [Paper]
LLM-QBench: A Benchmark Towards the Best Practice for Post-training Quantization of Large Language Models Arxiv 2024 [Paper] [Code]
SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models Arxiv 2024 [Paper]
Combining multiple post-training techniques to achieve most efficient quantized LLMs Arxiv 2024 [Paper]
Edge Intelligence Optimization for Large Language Model Inference with Batching and Quantization Arxiv 2024 [Paper]
SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models Arxiv 2024 [Paper] [Code]
OAC: Output-adaptive Calibration for Accurate Post-training Quantization Arxiv 2024 [Paper]
PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression Arxiv 2024 [Paper]
SpinQuant -- LLM quantization with learned rotations Arxiv 2024 [Paper]
Compressing Large Language Models using Low Rank and Low Precision Decomposition Arxiv 2024 [Paper] [Code]
Athena: Efficient Block-Wise Post-Training Quantization for Large Language Models Using Second-Order Matrix Derivative Information Arxiv 2024 [Paper]
Exploiting LLM Quantization Arxiv 2024 [Paper]
One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments Arxiv 2024 [Paper]
LCQ: Low-Rank Codebook based Quantization for Large Language Models Arxiv 2024 [Paper]
LoQT: Low Rank Adapters for Quantized Training Arxiv 2024 [Paper] [Code]
CLAQ: Pushing the Limits of Low-Bit Post-Training Quantization for LLMs Arxiv 2024 [Paper] [Code]
I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models Arxiv 2024 [Paper]
Outliers and Calibration Sets have Diminishing Effect on Quantization of Modern LLMs Arxiv 2024 [Paper]
DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs NeurIPS 2024 [Paper] [Code]
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization Arxiv 2024 [Paper] [Code]
Low-Rank Quantization-Aware Training for LLMs Arxiv 2024 [Paper]
TernaryLLM: Ternarized Large Language Model Arxiv 2024 [Paper]
Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark Arxiv 2024 [Paper] [Code]
Delta-CoMe: Training-Free Delta-Compression with Mixed-Precision for Large Language Models Arxiv 2024 [Paper]
QQQ: Quality Quattuor-Bit Quantization for Large Language Models Arxiv 2024 [Paper] [Code]
QTIP: Quantization with Trellises and Incoherence Processing NeurIPS 2024 [Paper] [Code]
Prefixing Attention Sinks can Mitigate Activation Outliers for Large Language Model Quantization EMNLP 2024 [Paper]
Mixture of Scales: Memory-Efficient Token-Adaptive Binarization for Large Language Models Arxiv 2024 [Paper]
Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization ISCA 2024 [Paper]
SDQ: Sparse Decomposed Quantization for LLM Inference Arxiv 2024 [Paper]
Attention-aware Post-training Quantization without Backpropagation Arxiv 2024 [Paper]
EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting Arxiv 2024 [Paper] [Code]
Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other Arxiv 2024 [Paper]
Layer-Wise Quantization: A Pragmatic and Effective Method for Quantizing LLMs Beyond Integer Bit-Levels Arxiv 2024 [Paper] [Code]
CDQuant: Accurate Post-training Weight Quantization of Large Pre-trained Models using Greedy Coordinate Descent Arxiv 2024 [Paper]
OutlierTune: Efficient Channel-Wise Quantization for Large Language Models Arxiv 2024 [Paper]
T-MAC: CPU Renaissance via Table Lookup for Low-Bit LLM Deployment on Edge Arxiv 2024 [Paper] [Code]
GPTQT: Quantize Large Language Models Twice to Push the Efficiency ICORIS 2024 [Paper]
Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment ACL 2024 [Paper]
How Does Quantization Affect Multilingual LLMs? EMNLP Findings 2024 [Paper]
RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization EMNLP Findings 2024 [Paper] [Code]
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients Arxiv 2024 [Paper] [Code]
FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision Arxiv 2024 [Paper] [Code]
Accuracy is Not All You Need Arxiv 2024 [Paper]
BitNet b1.58 Reloaded: State-of-the-art Performance Also on Smaller Networks Arxiv 2024 [Paper]
LeanQuant: Accurate Large Language Model Quantization with Loss-Error-Aware Grid Arxiv 2024 [Paper]
Fast Matrix Multiplications for Lookup Table-Quantized LLMs EMNLP Findings 2024 [Paper] [Code]
EfficientQAT: Efficient Quantization-Aware Training for Large Language Models Arxiv 2024 [Paper] [Code]
LRQ: Optimizing Post-Training Quantization for Large Language Models by Learning Low-Rank Weight-Scaling Matrices Arxiv 2024 [Paper] [Code]
Exploring Quantization for Efficient Pre-Training of Transformer Language Models EMNLP Findings 2024 [Paper] [Code]
Spectra: A Comprehensive Study of Ternary, Quantized, and FP16 Language Models Arxiv 2024 [Paper] [Code]
Mamba-PTQ: Outlier Channels in Recurrent Large Language Models Efficient Systems for Foundation Models Workshop @ ICML 2024 [Paper]
Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners Arxiv 2024 [Paper]
Accurate and Efficient Fine-Tuning of Quantized Large Language Models Through Optimal Balance Arxiv 2024 [Paper] [Code]
STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs Arxiv 2024 [Paper]
Advancing Multimodal Large Language Models with Quantization-Aware Scale Learning for Efficient Adaptation ACM MM 2024 [Paper]
ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models Arxiv 2024 [Paper]
MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models Arxiv 2024 [Paper] [Code (Marlin)] [Code (Sparse Marlin)]
Matmul or No Matmal in the Era of 1-bit LLMs Arxiv 2024 [Paper]
MobileQuant: Mobile-friendly Quantization for On-device Language Models EMNLP Findings 2024 [Paper] [Code]
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs Arxiv 2024 [Paper] [Code]
Foundations of Large Language Model Compression -- Part 1: Weight Quantization Arxiv 2024 [Paper] [Code]
OPAL: Outlier-Preserved Microscaling Quantization A ccelerator for Generative Large Language Models DAC 2024 [Paper]
VPTQ: Extreme Low-bit Vector Post-Training Quantization for Large Language Models EMNLP 2024 [Paper] [Code]
Scaling FP8 training to trillion-token LLMs Arxiv 2024 [Paper]
Accumulator-Aware Post-Training Quantization Arxiv 2024 [Paper]
Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores Arxiv 2024 [Paper]
Rotated Runtime Smooth: Training-Free Activation Smoother for accurate INT4 inference Arxiv 2024 [Paper] [Code]
EXAQ: Exponent Aware Quantization For LLMs Acceleration Arxiv 2024 [Paper]
ARB-LLM: Alternating Refined Binarizations for Large Language Models Arxiv 2024 [Paper] [Code]
PrefixQuant: Static Quantization Beats Dynamic through Prefixed Outliers in LLMs Arxiv 2024 [Paper] [Code]
SpaLLM: Unified Compressive Adaptation of Large Language Models with Sketching Arxiv 2024 [Paper]
Scaling Laws for Mixed quantization in Large Language Models Arxiv 2024 [Paper]
Q-VLM: Post-training Quantization for Large Vision-Language Models NeurIPS 2024 [Paper] [Code]
CrossQuant: A Post-Training Quantization Method with Smaller Quantization Kernel for Precise Large Language Model Compression Arxiv 2024 [Paper]
FlatQuant: Flatness Matters for LLM Quantization Arxiv 2024 [Paper] [Code]
DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization Arxiv 2024 [Paper]
QEFT: Quantization for Efficient Fine-Tuning of LLMs EMNLP Findings 2024 [Paper] [Code]
Continuous Approximations for Improving Quantization Aware Training of LLMs Arxiv 2024 [Paper]
DAQ: Density-Aware Post-Training Weight-Only Quantization For LLMs Arxiv 2024 [Paper]
COMET: Towards Partical W4A4KV4 LLMs Serving Arxiv 2024 [Paper]
Scaling laws for post-training quantized large language models Arxiv 2024 [Paper]
Channel-Wise Mixed-Precision Quantization for Large Language Models Arxiv 2024 [Paper]
Understanding the difficulty of low-precision post-training quantization of large language models Arxiv 2024 [Paper]
QuAILoRA: Quantization-Aware Initialization for LoRA NeurIPS Workshop on Efficient Natural Language and Speech Processing (ENLSP-IV) 2024 [Paper]
SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training NeurIPS 2024 [Paper]
Pyramid Vector Quantization for LLMs Arxiv 2024 [Paper]

Pruning and Sparsity

The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers ICLR 2023 [Paper]
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time ICML 2023 [Paper] [Code]
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation ICML 2023 [Paper] [Code]
LLM-Pruner: On the Structural Pruning of Large Language Models NeurIPS 2023 [Paper] [Code]
ZipLM: Inference-Aware Structured Pruning of Language Models NeurIPS 2023 [Paper] [Code]
H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models NeurIPS 2023 [Paper] [Code]
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter NeurIPS 2023 [Paper] [Code]
Learning to Compress Prompts with Gist Tokens NeurIPS 2023 [Paper]
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers NeurIPS 2023 [Paper]
Prune and Tune: Improving Efficient Pruning Techniques for Massive Language Models ICLR 2023 TinyPapers [Paper]
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot ICML 2023 [Paper] [Code]
AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning ICLR 2023 [Paper]
Rethinking the Role of Scale for In-Context Learning: An Interpretability-based Case Study at 66 Billion Scale ACL 2023 [Paper] [Code]
Structured Pruning for Efficient Generative Pre-trained Language Models ACL 2023 [Paper]
A Simple and Effective Pruning Approach for Large Language Models ICLR 2024 [Paper] [Code]
Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning ACL Findings 2024 [Paper]
Structural pruning of large language models via neural architecture search AutoML 2023 [Paper]
Pruning Large Language Models via Accuracy Predictor ICASSP 2024 [Paper]
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity VLDB 2024 [Paper] [Cde]
Compressing LLMs: The Truth is Rarely Pure and Never Simple ICLR 2024 [Paper]
Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs ICML 2024 [Paper] [Code]
Compresso: Structured Pruning with Collaborative Prompting Learns Compact Large Language Models Arxiv 2023 [Paper] [Code]
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Arxiv 2023 [Paper] [Code]
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning Arxiv 2023 [Paper] [Code]
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs ICLR 2024 [Paper] [Code]
One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models ICASSP 2024 [Paper]
Survival of the Most Influential Prompts: Efficient Black-Box Prompt Search via Clustering and Pruning EMNLP Findings 2023 [Paper]
The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models EMNLP Findings 2023 [Paper]
Divergent Token Metrics: Measuring degradation to prune away LLM components -- and optimize quantization Arxiv 2023 [Paper]
LoRAShear: Efficient Large Language Model Structured Pruning and Knowledge Recovery Arxiv 2023 [Paper]
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models Arxiv 2023 [Paper]
E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity Arxiv 2023 [Paper]
Beyond Size: How Gradients Shape Pruning Decisions in Large Language Models Arxiv 2023 [Paper] [Code]
On the Impact of Calibration Data in Post-training Quantization and Pruning ACL 2024 [Paper]
BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation OpenReview [Paper] [Code]
PUSHING GRADIENT TOWARDS ZERO: A NOVEL PRUNING METHOD FOR LARGE LANGUAGE MODELS OpenReview 2023 [Paper]
Plug-and-Play: An Efficient Post-training Pruning Method for Large Language Models ICLR 2024 [Paper] [Code]
Lighter, yet More Faithful: Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarization Arxiv 2023 [Paper] [Code]
LORAPRUNE: PRUNING MEETS LOW-RANK PARAMETER-EFFICIENT FINE-TUNING Arxiv 2023 [Paper]
Mini-GPTs: Efficient Large Language Models through Contextual Pruning Arxiv 2023 [Paper] [Code]
The LLM Surgeon Arxiv 2023 [Paper]
Fluctuation-based Adaptive Structured Pruning for Large Language Models AAAI 2024 [Paper]
How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark CPAL 2024 [Paper]
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs Arxiv 2023 [Paper]
Fast and Optimal Weight Update for Pruned Large Language Models Arxiv 2024 [Paper]
APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference Arxiv 2024 [Paper]
Scaling Sparse Fine-Tuning to Large Language Models Arxiv 2024 [Paper]
SliceGPT: Compress Large Language Models by Deleting Rows and Columns ICLR 2024 [Paper] [Code]
Shortened LLaMA: A Simple Depth Pruning for Large Language Models Arxiv 2024 [Paper]
Everybody Prune Now: Structured Pruning of LLMs with only Forward Passes Arxiv 2024 [Paper] [Code]
NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models Arxiv 2024 [Paper]
LaCo: Large Language Model Pruning via Layer Collapse EMNLP Findings 2024 [Paper]
Why Lift so Heavy? Slimming Large Language Models by Cutting Off the Layers Arxiv 2024 [Paper]
EBFT: Effective and Block-Wise Fine-Tuning for Sparse LLMs Arxiv 2024 [Paper] [Code]
Data-free Weight Compress and Denoise for Large Language Models Arxiv 2024 [Paper]
Gradient-Free Adaptive Global Pruning for Pre-trained Language Models Arxiv 2024 [Paper]
ShortGPT: Layers in Large Language Models are More Redundant Than You Expect Arxiv 2024 [Paper]
LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models Arxiv 2024 [Paper] [Code]
Compressing Large Language Models by Streamlining the Unimportant Layer Arxiv 2024 [Paper]
LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models Arxiv 2024 [Paper]
Shears: Unstructured Sparsity with Neural Low-rank Adapter Search NAACL 2024 [Paper]
Eigenpruning NAACL 2024 Abstract [Paper]
OpenBA-V2: Reaching 77.3% High Compression Ratio with Fast Multi-Stage Pruning Arxiv 2024 [Paper]
Pruning as a Domain-specific LLM Extractor NAACL 2024 Findings [Paper] [Code]
Differentiable Model Scaling using Differentiable Topk ICML 2024 [Paper]
COPAL: Continual Pruning in Large Language Generative Models ICML 2024 [Paper]
Pruner-Zero: Evolving Symbolic Pruning Metric from scratch for Large Language Models ICML 2024 [Paper] [Code]
Feature-based Low-Rank Compression of Large Language Models via Bayesian Optimization ACL Findings 2024 [Paper]
Surgical Feature-Space Decomposition of LLMs: Why, When and How? ACL 2024 [Paper]
Pruning Large Language Models to Intra-module Low-rank Architecture with Transitional Activations ACL Findings 2024 [Paper]
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning ACL Findings 2024 [Paper] [Code]
Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference ICML 2024 [Paper] [Code]
MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbations Arxiv 2024 [Paper] [Code]
ALPS: Improved Optimization for Highly Sparse One-Shot Pruning for Large Language Models Arxiv 2024 [Paper]
HiP Attention: Sparse Sub-Quadratic Attention with Hierarchical Attention Pruning Arxiv 2024 [Paper]
Optimization-based Structural Pruning for Large Language Models without Back-Propagation Arxiv 2024 [Paper]
BlockPruner: Fine-grained Pruning for Large Language Models Arxiv 2024 [Paper] [Code]
Rethinking Pruning Large Language Models: Benefits and Pitfalls of Reconstruction Error Minimization Arxiv 2024 [Paper]
RankAdaptor: Hierarchical Dynamic Low-Rank Adaptation for Structural Pruned LLMs Arxiv 2024 [Paper]
What Matters in Transformers? Not All Attention is Needed Arxiv 2024 [Paper] [Code]
Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging EMNLP 2024 [Paper]
ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models Arxiv 2024 [Paper] [Code]
Finding Transformer Circuits with Edge Pruning Arxiv 2024 [Paper] [Code]
Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs Arxiv 2024 [Paper] [Code]
MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models Arxiv 2024 [Paper]
Reconstruct the Pruned Model without Any Retraining Arxiv 2024 [Paper]
A deeper look at depth pruning of LLMs ICML TF2M Workshop 2024 [Paper] [Code]
Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining Arxiv 2024 [Paper]
Pruning Large Language Models with Semi-Structural Adaptive Sparse Training Arxiv 2024 [Paper]
A Convex-optimization-based Layer-wise Post-training Pruner for Large Language Models Arxiv 2024 [Paper]
ThinK: Thinner Key Cache by Query-Driven Pruning Arxiv 2024 [Paper]
LLM-Barber: Block-Aware Rebuilder for Sparsity Mask in One-Shot for Large Language Models Arxiv 2024 [Paper] [Code]
LLM Pruning and Distillation in Practice: The Minitron Approach Arxiv 2024 [Paper] [Models]
Training-Free Activation Sparsity in Large Language Models Arxiv 2024 [Paper]
PAT: Pruning-Aware Tuning for Large Language Models Arxiv 2024 [Paper] [Code]
Sirius: Contextual Sparsity with Correction for Efficient LLMs Arxiv 2024 [Paper] [Code]
STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning Arxiv 2024 [Paper]
Search for Efficient Large Language Models NeurIPS 2024 [Paper]
SlimGPT: Layer-wise Structured Pruning for Large Language Models NeurIPS 2024 [Paper]
Learn To be Efficient: Build Structured Sparsity in Large Language Models NeurIPS 2024 [Paper]
ALS: Adaptive Layer Sparsity for Large Language Models via Activation Correlation Assessment NeurIPS 2024 [Paper]
Getting Free Bits Back from Rotational Symmetries in LLMs Arxiv 2024 [Paper]
SLiM: One-shot Quantized Sparse Plus Low-rank Approximation of LLMs Arxiv 2024 [Paper] [Code]
Self-Data Distillation for Recovering Quality in Pruned Large Language Models NeurIPS 2024 Machine Learning and Compression Workshop [Paper]
EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search Arxiv 2024 [Paper] [Code]
Pruning Foundation Models for High Accuracy without Retraining EMNLP Findings 2024 [Paper] [Code]

Distillation

Lifting the Curse of Capacity Gap in Distilling Language Models ACL 2023 [Paper] [Code]
Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step ACL 2023 [Paper]
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes ACL 2023 [Paper]
SCOTT: Self-Consistent Chain-of-Thought Distillation ACL 2023 [Paper]
DISCO: Distilling Counterfactuals with Large Language Models ACL 2023 [Paper] [Code]
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions Arxiv 2023 [Paper] [Code]
How To Train Your (Compressed) Large Language Model Arxiv 2023 [Paper]
The False Promise of Imitating Proprietary LLMs Arxiv 2023 [Paper]
GPT4All: Training an Assistant-style Chatbot with Large Scale Data Distillation from GPT-3.5-Turbo Arxiv 2023 [Paper] [Code]
PaD: Program-aided Distillation Specializes Large Models in Reasoning Arxiv 2023 [Paper]
MiniLLM: Knowledge Distillation of Large Language Models ICLR 2024 [Paper] [Code]
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes ICLR 2024 [Paper]
GKD: Generalized Knowledge Distillation for Auto-regressive Sequence Models ICLR 2024 [Paper]
Chain-of-Thought Prompt Distillation for Multimodal Named Entity and Multimodal Relation Extraction Arxiv 2023 [Paper]
Task-agnostic Distillation of Encoder-Decoder Language Models Arxiv 2023 [Paper]
Sci-CoT: Leveraging Large Language Models for Enhanced Knowledge Distillation in Small Models for Scientific QA Arxiv 2023 [Paper]
Baby Llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty CoNLL 2023 [Paper] [Code]
Can a student Large Language Model perform as well as it's teacher? Arxiv 2023 [Paper]
Multistage Collaborative Knowledge Distillation from Large Language Models ACL 2024 [Paper] [Code]
Lion: Adversarial Distillation of Closed-Source Large Language Model EMNLP 2023 [Paper] [Code]
MCC-KD: Multi-CoT Consistent Knowledge Distillation EMNLP 2023 [Paper]
PromptMix: A Class Boundary Augmentation Method for Large Language Model Distillation EMNLP 2023 [Paper]
YODA: Teacher-Student Progressive Learning for Language Models Arxiv 2023 [Paper]
Knowledge Fusion of Large Language Models ICLR 2024 [Paper] [Code]
Knowledge Distillation for Closed-Source Language Models Arxiv 2024 [Paper]
TinyLLM: Learning a Small Student from Multiple Large Language Models Arxiv 2024 [Paper]
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Arxiv 2024 [Paper]
Revisiting Knowledge Distillation for Autoregressive Language Models ACL 2024 [Paper]
Sinkhorn Distance Minimization for Knowledge Distillation COLING 2024 [Paper]
Divide-or-Conquer? Which Part Should You Distill Your LLM? Arxiv 2024 [Paper]
Learning to Maximize Mutual Information for Chain-of-Thought Distillation ACL 2024 Findings [Paper]
DistiLLM: Towards Streamlined Distillation for Large Language Models ICML 2024 [Paper] [Code]
Efficiently Distilling LLMs for Edge Applications NAACL 2024 [Paper]
Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models Arxiv 2024 [Paper]
Distilling Algorithmic Reasoning from LLMs via Explaining Solution Programs Arxiv 2024 [Paper]
Direct Preference Knowledge Distillation for Large Language Models Arxiv 2024 [Paper] [Codes]
Dual-Space Knowledge Distillation for Large Language Models Arxiv 2024 [Paper] [Codes]
DDK: Distilling Domain Knowledge for Efficient Large Language Models Arxiv 2024 [Paper]
Compact Language Models via Pruning and Knowledge Distillation Arxiv 2024 [Paper] [Code]
LLM Pruning and Distillation in Practice: The Minitron Approach Arxiv 2024 [Paper] [Models]
The Mamba in the Llama: Distilling and Accelerating Hybrid Models Arxiv 2024 [Paper]
DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models EMNLP 2024 [Paper]

Efficient Prompting

Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning ACL 2023 [Paper] [Code]
Batch Prompting: Efficient Inference with Large Language Model APIs EMNLP 2023 [Paper] [Code]
Adapting Language Models to Compress Contexts EMNLP 2023 [Paper] [Code]
Compressing Context to Enhance Inference Efficiency of Large Language Models EMNLP 2023 [Paper] [Code]
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models EMNLP 2023 [Paper] [Code]
Vector-Quantized Prompt Learning for Paraphrase Generation EMNLP Findings 2023 [Paper]
Efficient Prompting via Dynamic In-Context Learning Arxiv 2023 [Paper]
Learning to Compress Prompts with Gist Tokens NeurIPS 2023 [Paper] [Code]
In-context Autoencoder for Context Compression in a Large Language Model ICLR 2024 [Paper]
Discrete Prompt Compression with Reinforcement Learning Arxiv 2023 [Paper] [Code]
BatchPrompt: Accomplish more with less Arxiv 2023 [Paper]
(Dynamic) Prompting might be all you need to repair Compressed LLMs Arxiv 2023 [Paper]
RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation Arxiv 2023 [Paper] [Code]
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression ACL 2023 [Paper] [Code]
Extending Context Window of Large Language Models via Semantic Compression Arxiv 2023 [Paper]
Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning EMNLP 2024 [Paper] [Code]
The Impact of Reasoning Step Length on Large Language Models ACL 2024 Findings [Paper]
Compressed Context Memory For Online Language Model Interaction ICLR 2024 [Paper] [Code]
Learning to Compress Prompt in Natural Language Formats Arxiv 2024 [Paper]
Say More with Less: Understanding Prompt Learning Behaviors through Gist Compression Arxiv 2024 [Paper] [Code]
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses Arxiv 2024 [Paper]
LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression Arxiv 2024 [Paper] [Code]
PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Models Arxiv 2024 [Paper] [Code]
PROMPT-SAW: Leveraging Relation-Aware Graphs for Textual Prompt Compression Arxiv 2024 [Paper]
Prompts As Programs: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization Arxiv 2024 [Paper] [Code]
Adapting LLMs for Efficient Context Processing through Soft Prompt Compression IPCA 2024 [Paper]
Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation Arxiv 2024 [Paper]
Unifying Demonstration Selection and Compression for In-Context Learning Arxiv 2024 [Paper]
SelfCP: Compressing Long Prompt to 1/12 Using the Frozen Large Language Model Itself Arxiv 2024 [Paper]
Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models Arxiv 2024 [Paper]
QUITO: Accelerating Long-Context Reasoning through Query-Guided Context Compression Arxiv 2024 [Paper] [Code]
500xCompressor: Generalized Prompt Compression for Large Language Models Arxiv 2024 [Paper]
Enhancing and Accelerating Large Language Models via Instruction-Aware Contextual Compression Arxiv 2024 [Paper]
Prompt Compression with Context-Aware Sentence Encoding for Fast and Improved LLM Inference Arxiv 2024 [Paper] [Code]
Learning to Compress Contexts for Efficient Knowledge-based Visual Question Answering Arxiv 2024 [Paper]
Parse Trees Guided LLM Prompt Compression Arxiv 2024 [Paper]
AlphaZip: Neural Network-Enhanced Lossless Text Compression Arxiv 2024 [Paper]
Discovering the Gems in Early Layers: Accelerating Long-Context LLMs with 1000x Input Token Reduction Arxiv 2024 [Paper] [Code]
Perception Compressor:A training-free prompt compression method in long context scenarios Arxiv 2024 [Paper]
From Reading to Compressing: Exploring the Multi-document Reader for Prompt Compression EMNLP Findings 2024 [Paper]
Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability EMNLP Findings 2024 [Paper]
Style-Compress: An LLM-Based Prompt Compression Framework Considering Task-Specific Styles EMNLP Findings 2024 [Paper]

KV Cache Compression

Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time NeurIPS 2023 [Paper]
Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs ICLR 2024 [Paper]
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization NeurIPS 2024 [Paper]
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache ICML 2024 [Paper] [Code]
No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization Arxiv 2024 [Paper]
Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference MLSys 2024 [Paper]
GEAR: An Efficient KV Cache Compression Recipefor Near-Lossless Generative Inference of LLM Arxiv 2024 [Paper]
QAQ: Quality Adaptive Quantization for LLM KV Cache Arxiv 2024 [Paper] [Code]
KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization Arxiv 2024 [Paper]
PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference ACL 2024 [Paper]
Unlocking Data-free Low-bit Quantization with Matrix Decomposition for KV Cache Compression Arxiv 2024 [Paper]
ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification Arxiv 2024 [Paper]
MiniCache: KV Cache Compression in Depth Dimension for Large Language Models Arxiv 2024 [Paper]
PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling Arxiv 2024 [Paper]
QJL: 1-Bit Quantized JL Transform for KV Cache Quantization with Zero Overhead Arxiv 2024 [Paper] [Code]
Effectively Compress KV Heads for LLM Arxiv 2024 [Paper]
A Simple and Effective L2 Norm-Based Strategy for KV Cache Compression EMNLP 2024 [Paper]
PQCache: Product Quantization-based KVCache for Long Context LLM Inference Arxiv 2024 [Paper]
Palu: Compressing KV-Cache with Low-Rank Projection Arxiv 2024 [Paper] [Code]
RazorAttention: Efficient KV Cache Compression Through Retrieval Heads Arxiv 2024 [Paper]
Finch: Prompt-guided Key-Value Cache Compression Arxiv 2024 [Paper]
Zero-Delay QKV Compression for Mitigating KV Cache and Network Bottlenecks in LLM Inference Arxiv 2024 [Paper]
Eigen Attention: Attention in Low-Rank Space for KV Cache Compression EMNLP Findings 2024 [Paper] [Code]
CSKV: Training-Efficient Channel Shrinking for KV Cache in Long-Context Scenarios Arxiv 2024 [Paper] [Code]
LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy Arxiv 2024 [Paper]
SimLayerKV: A Simple Framework for Layer-Level KV Cache Reduction Arxiv 2024 [Paper] [Code]
MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection Arxiv 2024 [Paper]
AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations Arxiv 2024 [Paper]
Residual vector quantization for KV cache compression in large language model Arxiv 2024 [Paper] [Code]
Lossless KV Cache Compression to 2% Arxiv 2024 [Paper]

Other

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness NeurIPS 2022 [Paper] [Code]
TensorGPT: Efficient Compression of the Embedding Layer in LLMs based on the Tensor-Train Decomposition Arxiv 2023 [Paper]
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers NeurIPS 2023 [Paper]
SkipDecode: Autoregressive Skip Decoding with Batching and Caching for Efficient LLM Inference Arxiv 2023 [Paper]
Scaling In-Context Demonstrations with Structured Attention Arxiv 2023 [Paper]
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline Arxiv 2023 [Paper] [Code]
CPET: Effective Parameter-Efficient Tuning for Compressed Large Language Models Arxiv 2023 [Paper]
Ternary Singular Value Decomposition as a Better Parameterized Form in Linear Mapping Arxiv 2023 [Paper]
LLMCad: Fast and Scalable On-device Large Language Model Inference Arxiv 2023 [Paper]
vLLM: Efficient Memory Management for Large Language Model Serving with PagedAttention Arxiv 2023 [Paper]
LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models Arxiv 2023 [Paper] [Code]
LORD: Low Rank Decomposition Of Monolingual Code LLMs For One-Shot Compression Arxiv 2023 [Paper] [Code]
Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation Arxiv 2023 [Paper]
Efficient Streaming Language Models with Attention Sinks Arxiv 2023 [Paper] [Code]
Efficient Large Language Models Fine-Tuning On Graphs Arxiv 2023 [Paper]
SparQ Attention: Bandwidth-Efficient LLM Inference Arxiv 2023 [Paper]
Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models Arxiv 2023 [Paper]
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU Arxiv 2023 [Paper] [Code]
Dataset Quantization ICCV 2023 [Paper] [Code]
Text Alignment Is An Efficient Unified Model for Massive NLP Tasks NeurIPS 2023 [Paper] [Code]
Context Compression for Auto-regressive Transformers with Sentinel Tokens EMNLP 2023 [Paper] [Code]
TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction EMNLP Findings 2023 [Paper]
Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression EMNLP Findings 2023 [Paper]
FFSplit: Split Feed-Forward Network For Optimizing Accuracy-Efficiency Trade-off in Language Model Inference Arxiv 2024 [Paper]
LoMA: Lossless Compressed Memory Attention Arxiv 2024 [Paper]
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Arxiv 2024 [Paper] [Code]
BiTA: Bi-Directional Tuning for Lossless Acceleration in Large Language Models Arxiv 2024 [Paper] [Code]
CompactifAI: Extreme Compression of Large Language Models using Quantum-Inspired Tensor Networks Arxiv 2024 [Paper]
MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases ICML 2024 [Paper] [Code]
BAdam: A Memory Efficient Full Parameter Training Method for Large Language Models Arxiv 2024 [Paper] [Code]
NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention Arxiv 2024 [Paper]
Not all Layers of LLMs are Necessary during Inference Arxiv 2024 [Paper]
GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection Arxiv 2024 [Paper]
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference Arxiv 2024 [Paper]
Smart-Infinity: Fast Large Language Model Training using Near-Storage Processing on a Real System HPCA 2024 [Paper]
ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Models Arxiv 2024 [Paper]
Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation Arxiv 2024 [Paper]
Training LLMs over Neurally Compressed Text Arxiv 2024 [Paper]
TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding Arxiv 2024 [Paper] [Code]
SnapKV: LLM Knows What You are Looking for Before Generation Arxiv 2024 [Paper] [Code]
Characterizing the Accuracy - Efficiency Trade-off of Low-rank Decomposition in Language Models Arxiv 2024 [Paper]
KV-Runahead: Scalable Causal LLM Inference by Parallel Key-Value Cache Generation ICML 2024 [Paper]
Token-wise Influential Training Data Retrieval for Large Language Models ACL 2024 [Paper] [Code]
Basis Selection: Low-Rank Decomposition of Pretrained Large Language Models for Target Applications Arxiv 2024 [Paper]
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference Arxiv 2024 [Paper]
AdaCoder: Adaptive Prompt Compression for Programmatic Visual Question Answering Arxiv 2024 [Paper]
CaM: Cache Merging for Memory-efficient LLMs Inference ICML 2024 [Paper] [Code]
CLLMs: Consistency Large Language Models ICML 2024 [Paper] [Code]
MoDeGPT: Modular Decomposition for Large Language Model Compression Arxiv 2024 [Paper]
Accelerating Large Language Model Training with Hybrid GPU-based Compression Arxiv 2024 [Paper]
Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models NeurIPS 2024 [Paper]
KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Head Arxiv 2024 [Paper]
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs EMNLP 2024 [Paper]
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Acceleration Arxiv 2024 [Paper] [Code]
UNComp: Uncertainty-Aware Long-Context Compressor for Efficient Large Language Model Inference Arxiv 2024 [Paper]
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression Arxiv 2024 [Paper] [Code]
Rodimus*: Breaking the Accuracy-Efficiency Trade-Off with Efficient Attentions Arxiv 2024 [Paper]
DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Heads Arxiv 2024 [Paper] [Code]
Progressive Mixed-Precision Decoding for Efficient LLM Inference Arxiv 2024 [Paper]

Tools

BMCook: Model Compression for Big Models [Code]
llama.cpp: Inference of LLaMA model in pure C/C++ [Code]
LangChain: Building applications with LLMs through composability [Code]
GPTQ-for-LLaMA: 4 bits quantization of LLaMA using GPTQ [Code]
Alpaca-CoT: An Instruction Fine-Tuning Platform with Instruction Data Collection and Unified Large Language Models Interface [Code]
vllm: A high-throughput and memory-efficient inference and serving engine for LLMs [Code]
LLaMA Efficient Tuning: Fine-tuning LLaMA with PEFT (PT+SFT+RLHF with QLoRA) [Code]
gpt-fast: Simple and efficient pytorch-native transformer text generation in <1000 LOC of python. [Code]
Efficient-Tuning-LLMs: (Efficient Finetuning of QLoRA LLMs). QLoRA, LLama, bloom, baichuan-7B, GLM [Code]
bitsandbytes: 8-bit CUDA functions for PyTorch [Code]
ExLlama: A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights. [Code]
lit-gpt: Hackable implementation of state-of-the-art open-source LLMs based on nanoGPT. Supports flash attention, 4-bit and 8-bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. [Code]
Lit-LLaMA: Implementation of the LLaMA language model based on nanoGPT. Supports flash attention, Int8 and GPTQ 4bit quantization, LoRA and LLaMA-Adapter fine-tuning, pre-training. [Code]
lama.onnx: LLaMa/RWKV onnx models, quantization and testcase [Code]
fastLLaMa: An experimental high-performance framework for running Decoder-only LLMs with 4-bit quantization in Python using a C/C++ backend. [Code]
Sparsebit: A model compression and acceleration toolbox based on pytorch. [Code]
llama2.c: Inference Llama 2 in one file of pure C [Code]
Megatron-LM: Ongoing research training transformer models at scale [Code]
ggml: Tensor library for machine learning [Code]
LLamaSharp: C#/.NET binding of llama.cpp, including LLaMa/GPT model inference and quantization, ASP.NET core integration and UI [Code]
rwkv.cpp: NT4/INT5/INT8 and FP16 inference on CPU for RWKV language model [Code]
Can my GPU run this LLM?: Calculate GPU memory requirement & breakdown for training/inference of LLM models. Supports ggml/bnb quantization [Code]
TinyChatEngine: On-Device LLM Inference Library [Code]
TensorRT-LLM: TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. [Code]
IntLLaMA: A fast and light quantization solution for LLaMA [Code]
EasyLLM: Built upon Megatron-Deepspeed and HuggingFace Trainer, EasyLLM has reorganized the code logic with a focus on usability. While enhancing usability, it also ensures training efficiency [Code]
GreenBit LLaMA: Advanced Ultra-Low Bitrate Compression Techniques for the LLaMA Family of LLMs [Code]
Intel® Neural Compressor: An open-source Python library supporting popular model compression techniques on all mainstream deep learning frameworks (TensorFlow, PyTorch, ONNX Runtime, and MXNet) [Code]
LLM-Viewer: Analyze the inference of Large Language Models (LLMs). Analyze aspects like computation, storage, transmission, and hardware roofline model in a user-friendly interface. [Code]
LLaMA3-Quantization: A repository dedicated to evaluating the performance of quantizied LLaMA3 using various quantization methods. [Code]
LLamaSharp: A C#/.NET library to run LLM models (🦙LLaMA/LLaVA) on your local device efficiently. [Code]
Green-bit-LLM: A toolkit for fine-tuning, inferencing, and evaluating GreenBitAI's LLMs. [Code] [Model]
Bitorch Engine: Streamlining AI with Open-Source Low-Bit Quantization. [Code]
llama-zip: LLM-powered lossless compression tool [Code]
LLaMA-Factory: Unify Efficient Fine-Tuning of 100+ LLMs [Code]
LLMC: A tool designed for LLM Compression. [Code]
BitBLAS: BitBLAS is a library to support mixed-precision matrix multiplications, especially for quantized LLM deployment. [Code]
AutoFP8: Open-source FP8 quantization library for producing compressed checkpoints for running in vLLM [Code]
AutoGGUF: automatically quant GGUF models [Code]
Transformer Compression: For releasing code related to compression methods for transformers, accompanying our publications [Code]

Contributing

This is an active repository and your contributions are always welcome! Before you add papers/tools into the awesome list, please make sure that:

The paper or tools is related to Large Language Models (LLMs). If the compression algorithms or tools are only evaluated on small-scale language models (e.g., BERT), they should not be included in the list.
The paper should be inserted in the correct position in chronological order (publication/arxiv release time).
The link to [Paper] should be the arxiv page, not the pdf page if this is a paper posted on arxiv.
If the paper is accpeted, please use the correct publication venue instead of arxiv

Thanks again for all the awesome contributors to this list!