Awesome

📘Documentation | 🛠️Quick Start | 🚀Algorithms | 👀Evaluation | 🤔Reporting Issues

Our 100K Instruction-Following Datasets

Align-Anything aims to align any modality large models (any-to-any models), including LLMs, VLMs, and others, with human intentions and values. More details about the definition and milestones of alignment for Large Models can be found in AI Alignment. Overall, this framework has the following characteristics:

• Highly Modular Framework. Its versatility stems from the abstraction of different algorithm types and well-designed APIs, allowing users to easily modify and customize the code for different tasks.
• Support for Various Model Fine-Tuning. This framework includes fine-tuning capabilities for models such as LLaMA3.1, LLaVA, Gemma, Qwen, Baichuan, and others (see Model Zoo).
• Support Fine-Tuning across Any Modality. It supports fine-tuning alignments for different modality model, including LLMs, VLMs, and other modalities (see Development Roadmap).
• Support Different Alignment Methods. The framework supports different alignment algorithms, including SFT, DPO, PPO, and others.

Note: We provide a quick start guide for users to quickly get the code structure and development details.

Alignment fine-tuning can significantly enhance the instruction-following capabilities of large multimodal models. After fine-tuning, Chameleon 7B Plus generates images that are more relevant to the prompt.

Quick Start

Easy Installation

# clone the repository
git clone git@github.com:PKU-Alignment/align-anything.git
cd align-anything

# create virtual env
conda create -n align-anything python==3.11
conda activate align-anything

• [Optional] We recommend installing CUDA in the conda environment and set the environment variable.

# We tested on the H800 computing cluster, and this version of CUDA works well.
# You can adjust this version according to the actual situation of the computing cluster.

conda install nvidia/label/cuda-12.2.0::cuda
export CUDA_HOME=$CONDA_PREFIX

If your CUDA installed in a different location, such as /usr/local/cuda/bin/nvcc, you can set the environment variables as follows:

export CUDA_HOME="/usr/local/cuda"

Finally, install align-anything by:

# We prepare quick installation for training and evaluation.
# If you only need to use the training or evaluation module,
# you can install the corresponding dependencies.
pip install -e .[train] # install the training dependencies
pip install -e .[evaluate] # install the evaluation dependencies

# If you need to install all dependencies, you can use the following command:
pip install -e .[all]

• pip install -e .[text-to-audio]: Install the text-to-audio dependencies.
• pip install -e .[minicpmv]: Install the minicpmv dependencies.

Training

We provide some scripts for quick start, you can find them in the ./scripts directory. These scripts would automatically download the model and dataset, and run the training or evaluation.

For example, scripts/llava_dpo.sh is the script for Text + Image -> Text modality, you can run it by:

cd scripts
bash llava_dpo.sh

Evaluation

After training, you can evaluate the model by running the scripts/llava_eval.sh script.

cd scripts
bash llava_eval.sh

You can simply modify the parameters in the script to suit your needs, e.g., the MODEL_NAME_OR_PATH for your own model or TRAIN_DATASETS for your own dataset. For more details please refer to the Advanced Usage section.

Algorithms

We support basic alignment algorithms for different modalities, each of which may involve additional algorithms. For instance, in the text modality, we have also implemented SimPO, KTO, and others.

Evaluation

We support evaluation datasets for Text -> Text, Text+Image -> Text and Text -> Image.

• ⚒️ : coming soon.

Wandb Logger

We support wandb logging. By default, it is set to offline. If you need to view wandb logs online, you can specify the environment variables of WANDB_API_KEY before starting the training:

export WANDB_API_KEY="..."  # your W&B API key here

Advanced Usage

Training

Q (Training Model Registration): What models are supported for training? What should I pay attention to if I want to use my own model?

A: The models registration of align-anything is 2 folds:

1. The model has been manually supported by the align-anything team, they are:

Besides, you can also use your own model for training, you can refer to the here (sorry, corresponding docs will be uploaded later) for the model registration.

Q (Training Dataset Registration): What datasets are supported for training? What should I pay attention to if I want to use my own dataset?

A: We prepare datasets_formatter for dataset registration. Its core function is to mapping the dataset key to conversation format.

Basically, we support 3 types of dataset format:

We introduce the following example below, and you can refer to here for more details.

• format_supervised_sample:

@register_template('Alpaca')
class Alpaca(BaseFormatter):

    def format_supervised_sample(self, raw_sample: dict[str, Any]) -> tuple[list[dict[str, Any]], dict]:
        prompt = ' '.join((raw_sample['instruction'], raw_sample['input']))
        response = raw_sample['output']
        return [
            {"role": "user", "content": prompt},
            {"role": "assistant", "content": response},
        ], {}

• format_preference_sample:

@register_template('AA_TI2T')
class AA_TI2T(BaseFormatter):
    system_prompt: str = ""

    def format_preference_sample(self, raw_sample: dict[str, Any]) -> tuple[list[dict[str, Any]], list[dict[str, Any]], dict[str, Any]]:
        better_id = int(raw_sample['overall_response'])
        worse_id = 2 if better_id==1 else 1

        if better_id not in [1, 2] or worse_id not in [1, 2]:
            return [], [], {}

        raw_better_response = raw_sample[f'response_{better_id}']
        raw_worse_response = raw_sample[f'response_{worse_id}']
        prompt = raw_sample['question']
        image = raw_sample['image'].convert('RGBA')
        better_conversation = [
            {'role': 'user', 'content': [
                    {'type': 'image'},
                    {'type': 'text', 'text': prompt},
                ]
            },
            {'role': 'assistant', 'content': [{'type': 'text', 'text': raw_better_response}]},
        ]
        worse_conversation = [
            {'role': 'user', 'content': [
                    {'type': 'image'},
                    {'type': 'text', 'text': prompt},
                ]
            },
            {'role': 'assistant', 'content': [{'type': 'text', 'text': raw_worse_response}]},
        ]

        meta_info = {
            'image': image,
            'better_response': raw_better_response,
            'worse_response': raw_worse_response,
        }

        return better_conversation, worse_conversation, meta_info

• format_prompt_only_sample:

@register_template('AA_TA2T')
class AA_TA2T(BaseFormatter):
    system_prompt: str = 'You are a helpful assistant.'

    def format_prompt_only_sample(self, raw_sample: dict[str, Any]) -> dict[str, Any]:
        prompt = raw_sample['prompt']
        audio_path = raw_sample['audio_path']

        conversation = [
            {'role': 'system', 'content': [{'type': 'text', 'text': self.system_prompt}]},
            {'role': 'user', 'content': [
                    {'type': 'audio', 'audio_url': audio_path},
                    {'type': 'text', 'text': prompt},
                ]},
        ]

        return conversation, {'audio_path': audio_path}

Evaluation

Q (Evaluation Model Registration): What models are supported for evaluation? What should I pay attention to if I want to use my own model?

A: Register your model to use align-anything for evaluation is easy, you only need to add your model special token to the ./align_anything/configs/eval_template.py file.

For example, if you want to use liuhaotian/llava-v1.5-7b for evaluation, you need to add the following template for it to the ./align_anything/configs/eval_template.py file:

@register_template('Llava')
class Llava:
    system_prompt: str = ''
    user_prompt: str = 'USER: \n<image>{input}'
    assistant_prompt: str = '\nASSISTANT:{output}'
    split_token: str = 'ASSISTANT:'
    separator: str = '###'

Evaluation

All evaluation scripts can be found in the ./scripts. The ./scripts/evaluate.sh script runs model evaluation on the benchmarks, and parameters that require user input have been left empty. The corresponding script is as follow:

You can modify the configuration files for the benchmarks in this directory to suit specific evaluation tasks and models, and adjust inference parameters for vLLM or DeepSpeed based on your generation backend. For more details about the evaluation pipeline, refer to the here.

Inference

Interactive Client

python3 -m align_anything.serve.cli --model_name_or_path your_model_name_or_path

Interactive Arena

python3 -m align_anything.serve.arena \
    --red_corner_model_name_or_path your_red_model_name_or_path \
    --blue_corner_model_name_or_path your_blue_model_name_or_path

Report Issues

If you have any questions in the process of using align-anything, don't hesitate to ask your questions on the GitHub issue page, we will reply to you in 2-3 working days.

Citation

Please cite the repo if you use the data or code in this repo.

@misc{align_anything,
  author = {PKU-Alignment Team},
  title = {Align Anything: training all modality models to follow instructions with unified language feedback},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/PKU-Alignment/align-anything}},
}

License

align-anything is released under Apache License 2.0.