Home

Awesome

<a name="readme-top"></a>

Liger Kernel: Efficient Triton Kernels for LLM Training

<table style="width: 100%; text-align: center; border-collapse: collapse;"> <tr> <th style="padding: 10px;" colspan="2">Stable</th> <th style="padding: 10px;" colspan="2">Nightly</th> <th style="padding: 10px;">Discord</th> <th style="padding: 10px;">Gurubase (experimental)</th> </tr> <tr> <td style="padding: 10px;"> <a href="https://pepy.tech/project/liger-kernel"> <img src="https://static.pepy.tech/badge/liger-kernel" alt="Downloads (Stable)"> </a> </td> <td style="padding: 10px;"> <a href="https://pypi.org/project/liger-kernel"> <img alt="PyPI - Version" src="https://img.shields.io/pypi/v/liger-kernel?color=green"> </a> </td> <td style="padding: 10px;"> <a href="https://pepy.tech/project/liger-kernel-nightly"> <img src="https://static.pepy.tech/badge/liger-kernel-nightly" alt="Downloads (Nightly)"> </a> </td> <td style="padding: 10px;"> <a href="https://pypi.org/project/liger-kernel-nightly"> <img alt="PyPI - Version" src="https://img.shields.io/pypi/v/liger-kernel-nightly?color=green"> </a> </td> <td style="padding: 10px;"> <a href="https://discord.gg/gpumode"> <img src="https://dcbadge.vercel.app/api/server/gpumode?style=flat" alt="Join Our Discord"> </a> </td> <td style="padding: 10px;"> <a href="https://gurubase.io/g/liger-kernel"> <img src="https://img.shields.io/badge/Gurubase-Ask%20Liger%20Kernel%20Guru-006BFF" alt="Ask Liger Kernel Guru"> </a> </td> </tr> </table> <img src="https://raw.githubusercontent.com/linkedin/Liger-Kernel/main/docs/images/logo-banner.png">

Installation | Getting Started | Examples | APIs | Cite our work

<details> <summary>Latest News 🔥</summary> </details>

Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduces memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come. The kernel works out of the box with Flash Attention, PyTorch FSDP, and Microsoft DeepSpeed. We welcome contributions from the community to gather the best kernels for LLM training.

Supercharge Your Model with Liger Kernel

Banner

With one line of code, Liger Kernel can increase throughput by more than 20% and reduce memory usage by 60%, thereby enabling longer context lengths, larger batch sizes, and massive vocabularies.

Speed UpMemory Reduction
Speed upMemory

Note:

Examples

Use CaseDescription
Hugging Face TrainerTrain LLaMA 3-8B ~20% faster with over 40% memory reduction on Alpaca dataset using 4 A100s with FSDP
Lightning TrainerIncrease 15% throughput and reduce memory usage by 40% with LLaMA3-8B on MMLU dataset using 8 A100s with DeepSpeed ZeRO3
Medusa Multi-head LLM (Retraining Phase)Reduce memory usage by 80% with 5 LM heads and improve throughput by 40% using 8 A100s with FSDP
Vision-Language Model SFTFinetune Qwen2-VL on image-text data using 4 A100s with FSDP

Key Features

Installation

Dependencies

CUDA

ROCm

Optional Dependencies

Note: Our kernels inherit the full spectrum of hardware compatibility offered by Triton.

To install the stable version:

$ pip install liger-kernel

To install the nightly version:

$ pip install liger-kernel-nightly

To install from source:

git clone https://github.com/linkedin/Liger-Kernel.git
cd Liger-Kernel
pip install -e .
# or if using transformers
pip install -e .[transformers]

Getting Started

There are a couple of ways to apply Liger kernels, depending on the level of customization required.

1. Use AutoLigerKernelForCausalLM

Using the AutoLigerKernelForCausalLM is the simplest approach, as you don't have to import a model-specific patching API. If the model type is supported, the modeling code will be automatically patched using the default settings.

from liger_kernel.transformers import AutoLigerKernelForCausalLM

# This AutoModel wrapper class automatically monkey-patches the
# model with the optimized Liger kernels if the model is supported.
model = AutoLigerKernelForCausalLM.from_pretrained("path/to/some/model")

2. Apply Model-Specific Patching APIs

Using the patching APIs, you can swap Hugging Face models with optimized Liger Kernels.

import transformers
from liger_kernel.transformers import apply_liger_kernel_to_llama

# 1a. Adding this line automatically monkey-patches the model with the optimized Liger kernels
apply_liger_kernel_to_llama()

# 1b. You could alternatively specify exactly which kernels are applied
apply_liger_kernel_to_llama(
  rope=True,
  swiglu=True,
  cross_entropy=True,
  fused_linear_cross_entropy=False,
  rms_norm=False
)

# 2. Instantiate patched model
model = transformers.AutoModelForCausalLM("path/to/llama/model")

3. Compose Your Own Model

You can take individual kernels to compose your models.

from liger_kernel.transformers import LigerFusedLinearCrossEntropyLoss
import torch.nn as nn
import torch

model = nn.Linear(128, 256).cuda()

# fuses linear + cross entropy layers together and performs chunk-by-chunk computation to reduce memory
loss_fn = LigerFusedLinearCrossEntropyLoss()

input = torch.randn(4, 128, requires_grad=True, device="cuda")
target = torch.randint(256, (4, ), device="cuda")

loss = loss_fn(model.weight, input, target)
loss.backward()

APIs

AutoModel

AutoModel VariantAPI
AutoModelForCausalLMliger_kernel.transformers.AutoLigerKernelForCausalLM

Patching

ModelAPISupported Operations
LLaMA 2 & 3liger_kernel.transformers.apply_liger_kernel_to_llamaRoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy
LLaMA 3.2-Visionliger_kernel.transformers.apply_liger_kernel_to_mllamaRoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Mistralliger_kernel.transformers.apply_liger_kernel_to_mistralRoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Mixtralliger_kernel.transformers.apply_liger_kernel_to_mixtralRoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Gemma1liger_kernel.transformers.apply_liger_kernel_to_gemmaRoPE, RMSNorm, GeGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Gemma2liger_kernel.transformers.apply_liger_kernel_to_gemma2RoPE, RMSNorm, GeGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Qwen2 & Qwen2.5liger_kernel.transformers.apply_liger_kernel_to_qwen2RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Qwen2-VLliger_kernel.transformers.apply_liger_kernel_to_qwen2_vlRMSNorm, LayerNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy
Phi3 & Phi3.5liger_kernel.transformers.apply_liger_kernel_to_phi3RoPE, RMSNorm, SwiGLU, CrossEntropyLoss, FusedLinearCrossEntropy

Kernels

KernelAPI
RMSNormliger_kernel.transformers.LigerRMSNorm
LayerNormliger_kernel.transformers.LigerLayerNorm
RoPEliger_kernel.transformers.liger_rotary_pos_emb
SwiGLUliger_kernel.transformers.LigerSwiGLUMLP
GeGLUliger_kernel.transformers.LigerGEGLUMLP
CrossEntropyliger_kernel.transformers.LigerCrossEntropyLoss
FusedLinearCrossEntropyliger_kernel.transformers.LigerFusedLinearCrossEntropyLoss
KLDivergenceliger_kernel.transformers.LigerKLDIVLoss
JSDliger_kernel.transformers.LigerJSD
FusedLinearJSDliger_kernel.transformers.LigerFusedLinearJSD
<!-- TODO: verify vocab sizes are accurate -->

Experimental Kernels

KernelAPI
Embeddingliger_kernel.transformers.experimental.LigerEmbedding
Matmul int2xint8liger_kernel.transformers.experimental.matmul
<!-- TODO: be more specific about batch size -->

Contributing, Acknowledgements, and License

Contact

Cite this work

Biblatex entry:

@article{hsu2024ligerkernelefficienttriton,
      title={Liger Kernel: Efficient Triton Kernels for LLM Training},
      author={Pin-Lun Hsu and Yun Dai and Vignesh Kothapalli and Qingquan Song and Shao Tang and Siyu Zhu and Steven Shimizu and Shivam Sahni and Haowen Ning and Yanning Chen},
      year={2024},
      eprint={2410.10989},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2410.10989},
      journal={arXiv preprint arXiv:2410.10989},
}

Star History

Star History Chart

Contributors

<a href="https://github.com/linkedin/Liger-Kernel/graphs/contributors"> <img alt="contributors" src="https://contrib.rocks/image?repo=linkedin/Liger-Kernel"/> </a> <p align="right" style="font-size: 14px; color: #555; margin-top: 20px;"> <a href="#readme-top" style="text-decoration: none; color: #007bff; font-weight: bold;"> ↑ Back to Top ↑ </a> </p>