Home

Awesome

SPPO: Self-Play Preference Optimization for Language Model Alignment

Mistral-7B-Instruct Llama-3-8B-Instruct AlpacaEval 2.0 Open LLM MT-Bench

This repository contains the official code and released models for the paper Self-Play Preference Optimization for Language Model Alignment.

Authors: Yue Wu*, Zhiqing Sun*, Huizhuo Yuan*, Kaixuan Ji, Yiming Yang, Quanquan Gu

[Webpage] [Huggingface] [Paper]

🔔 News

Table of Content

About SPPO

We propose a new self-play framework dubbed SPPO for language model alignment and a new learning objective (called SPPO loss) derived from the self-play framework to fine-tune large language models efficiently.

<p align="center"> <img src="images/table.png" width="65%"> <br> AlpacaEval 2.0 leaderboard results of normal and length-controlled (LC) win rates in percentage (\%). Mistral-7B-SPPO can outperform larger models and Mistral-7B-SPPO (best-of-16) can outperform proprietary models such as GPT-4(6/13). Llama-3-8B-SPPO exhibits even better performance. </p>

SPPO can significantly enhance the performance of an LLM without strong external signals such as responses or preferences from GPT-4. It can outperform the model trained with iterative direct preference optimization (DPO), among other methods. SPPO is theoretically grounded, ensuring that the LLM can converge to the von Neumann winner (i.e., Nash equilibrium) under general, potentially intransitive preference, and empirically validated through extensive evaluations on multiple datasets.

For more details, you can check our paper here.

Base Models and Released Models

ModelAlpacaEval2.0 LC Win RateAlpacaEval2.0 Win Rate
🤗Mistral-7B-Instruct-v0.217.1114.72
🤗Mistral-7B-SPPO Iter124.7923.51
🤗Mistral-7B-SPPO Iter226.8927.62
🤗Mistral-7B-SPPO Iter328.5331.02
🤗Llama-3-8B-Instruct22.9222.57
🤗Llama-3-8B-SPPO Iter131.7331.74
🤗Llama-3-8B-SPPO Iter235.1535.98
🤗Llama-3-8B-SPPO Iter338.7739.85
🤗Gemma-2-9B-It45.0835.62
🤗Gemma-2-9B-SPPO Iter148.7040.76
🤗Gemma-2-9B-SPPO Iter250.9344.64
🤗Gemma-2-9B-SPPO Iter353.2747.74

Environment Setup

Our training code is based on the alignment-handbook codebase. We utilize vllm for generation and pairRM for ranking. Follow the steps below to set up your environment:

  1. Create a Virtual Environment:

    conda create -n sppo python=3.10
conda activate sppo

  2. Install vllm for Generation:

    pip install vllm

  3. Install PairRM:

    git clone https://github.com/yuchenlin/LLM-Blender.git
cd LLM-Blender
pip install -e .

  4. Download and Install Training Dependencies:

    git clone https://github.com/uclaml/SPPO.git
cd SPPO
pip install -e .

Training Scripts

Execute the training scripts based on the base model you choose:

These scripts manage the training iterations, generation, and PairRM ranking processes. Note that some scripts may attempt to push datasets to the Hugging Face Hub under the UCLA-AGI organization. Ensure you have write access, or modify the organization name accordingly, or comment out any push_to_hub commands if necessary. Detailed scripts for each component are listed as follows:

Breakdown of Scripts:

  1. Generation: 
    python scripts/generate.py --model $MODEL --maxlen 2048 --output_dir $OUTPUT_DIR --prompts $PROMPTS

Main parameters:

  1. Ranking: 
    python scripts/rank.py --output_dir $OUTPUT_DIR --prompts $PROMPTS

Main Parameters:

  1. Training: 
    bash scripts/pipeline.sh --model $MODEL --iter $ITER --dataset $DATASET --output_dir $OUTPUT_DIR --num 1

Main Parameters:

Evaluation

We adhere to the established guidelines for evaluation and utilize the following repositories:

We provide the model configurations used during AlpacaEval 2 in the models_configs directory. Please note that after the initial release of our model, we retrained it using a slightly modified prompt. The win rates observed post-retraining are comparable to the original results.

Troubleshoot

For questions related to the paper, please contact the authors via email. If you encounter any issues with the code or wish to report a bug, feel free to open an issue on our GitHub repository.

Citation

@article{wu2024self,
  title={Self-play preference optimization for language model alignment},
  author={Wu, Yue and Sun, Zhiqing and Yuan, Huizhuo and Ji, Kaixuan and Yang, Yiming and Gu, Quanquan},
  year={2024}
}

Acknowledgements

We thank the authors of The Alignment Handbook for their foundational contributions to the training code. We also acknowledge the use of PairRM for ranking and vllm for generation.