Home

Awesome

Large Language Model Codebase

This repository is a framework for beginners to dive into llm training, inference and evaluation. The repository is based on huggingface transformers.

Overview

Training

ModelSupported training method
bertClassification, Reward Model Training
llamaReward Model Training, Supervised Finetuning, Rejection Sampling, RRHF, Weighted Learning

Inference

Evaluation

Installation

First, you should use the following command to clone the repository.

git clone https://github.com/underwoodnoble/llm_codebase.git
cd llm_codebase

For most of the tasks, use the following command to install the required packages.

pip install -r requirements/normal.txt

For DPO, use the following command to install the required packages.

pip install -r requirements/dpo.txt

For llama1 reward model, use the following command to install the required packages.

pip install -r requirements/llama_rm.txt

Classification

There are two ways to train a classification model in the repository. The first one is to train a model with a classification head. The second one is to train a model with a classification dict head. The classification dict head is a head that can be used to train a model with multiple classification tasks.

Single objective classification

Supported Models: Bert, Llama

Under the hood, the classification head is a linear layer on top of the model output. This repository uses transformers AutoModelForSequenceClassification to train a classification model. The following code shows how to train a classification model with a classification head.

Specific Arguments

Data Format

{
    "text": "text",
    "label": "label"
}

Bert

bash scripts/bert_classification.sh

Llama

You can use accelerate or deepspeed to speed up the training process.

Accelerate
REPO_DIR=repo_dir
DATA_DIR=data_dir
DeepSpeed

Multi-objective classification

Reward Model Training

Supported Models: Bert, Llama Specific Arguments

Data Format

{
    "text": ["text1", "text2"],
    "score": ["score1", "score2"]
}

Bert reward model

bash scripts/bert_reward.sh

Llama reward model

For Llama1, we recommend to use the requirements/llama1.txt to install the required packages. Because we find that higher version of transformers may degrade the preformance of Llama1. And do not use bf16 to train llama reward model, this will degrade the performance too.

# Install the required packages
pip install -r requirements/llama1.txt
# Train the reward model
bash scripts/llama1_reward.sh