Awesome

NeurIPS 2023 (Spotlight)

Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision

Introduction

Dromedary is an open-source self-aligned language model trained with minimal human supervision. For comprehensive details and insights, we kindly direct you to our project page and paper.

Update: Dromedary-2 (SFT)

The new SELF-ALIGN process in Dromedary-2 only involves two stages, We replace the first stage with diverse user prompts from ShareGPT, Dolly-15k, OpenAssistant, and OpenOrca, and create an improved prompt with one additional exemplar that encourages the LLM AI-assistant to generate responses in a general-specific-general response style, i.e., initiate with an overview, delve into specifics, and wrap up with a summary. Specifically, we directly take the one-shot exemplar from FastChat as this additional exemplar.

By utilizing the new principle-driven self-alignment prompt, we found that the LLaMA-2 base model with the improved ICL exemplars can achieve enhanced performance even without the verbose cloning phase nor inference-time few-shot examples. Therefore, we also drop the last stage of the original SELF-ALIGN process.

Dromedary-2 (RLAIF)

The SALMON (Self-ALignMent with principle-fOllowiNg reward models) training pipeline of Dromedary-2 can be found in the IBM/SALMON repository.

Original Dromedary

The repo for the original Dromedary release is in the dromedary_v1 branch.

Setup

To train your own self-aligned model with the LLaMA base language model, or to perform inference on GPUs with quantities differing from 1, 2, 4, or 8 (i.e., any power of 2), you should install our customized llama_dromedary package.

In a conda env with pytorch / cuda available, run:

cd llama_dromedary
pip install -r requirements.txt
pip install -e .
cd ..

Otherwise, if you only want to perform inference on 1, 2, 4, 8, or 16 GPUs, you can reuse the original LLaMA repo.

git clone https://github.com/facebookresearch/llama.git
cd llama
pip install -r requirements.txt
pip install -e .
cd ..

In addition, you should at least install the packages required for inference:

cd inference
pip install -r requirements.txt

Model Weights

We release Dromedary weights as delta weights to comply with the LLaMA model license. You can add our delta to the original LLaMA weights to obtain the Dromedary weights. Instructions:

1. Get the original LLaMA weights in the Hugging Face format by following the instructions here.
2. Download the LoRA delta weights from our Hugging Face model hub.
3. Follow our inference guide to see how to deploy Dromedary/LLaMA on your own machine with model parallel (which should be significantly faster than Hugging Face's default pipeline parallel when using multiple GPUs).

Synthetic Data for Self-Align

We release the synthetic data used to train Dromedary-65b (final) in Hugging Face Datasets Hub here.

The instructions are generated by the base LLaMA model with the (Topic-Guided Red-Teaming) Self-Instruct framework, while the responses are generated by the Dromedary (non-verbose) model prompted with the verbose prompt.

Update: We also release the synthetic data used to train Dromedary-2-70b (SFT) in Hugging Face Datasets Hub here.

Inference

We provide a chatbot demo for Dromedary.

Training

We provide the full training pipeline of Dromedary for reproduction.

Prompts

All the human annotations used in this project can be found here.

Citation

Please cite the following paper if you use the data or code in this repo.

@inproceedings{sun2023principle,
      title = {Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision},
      author = {Sun, Zhiqing and Shen, Yikang and Zhou, Qinhong and Zhang, Hongxin and Chen, Zhenfang and Cox, David and Yang, Yiming and Gan, Chuang},
      booktitle = {Thirty-seventh Conference on Neural Information Processing Systems},
      year = {2023},
      url = {https://openreview.net/forum?id=p40XRfBX96},
}

Acknowledgements

We thank Yizhong Wang for providing the code for the parse analysis plot. We also thank Meta LLaMA team, Standford Alpaca team, Vicuna team, Alpaca-LoRA, QLoRA team, and Hugging Face PEFT for their open-source efforts in democratizing large language models.