Awesome

Table of Contents

1. News
2. Installation
3. Uasge
   • List of Recipes
   • Configuration Priority
4. Features
5. Acknowledge
6. Citation

News

• [Update Jun. 12, 2024] Recipes for MaLa-ASR has been supported.
• [CALL FOR EXAMPLE] We sincerely invite developers and researchers to develop new applications, conduct academic research based on SLAM-LLM, and pull request your examples! We also acknowledge engineering PR (such as improving and speeding up multi-node training).
• [Update May. 22, 2024] Please join slack or WeChat group. We will sync our updates and Q&A here.
• [Update May. 21, 2024] Recipes for Spatial Audio Understanding has been supported.
• [Update May. 20, 2024] Recipes for music caption (MC) has been supported.
• [Update May. 8, 2024] Recipes for visual speech recognition (VSR) has been supported.
• [Update May. 4, 2024] Recipes for zero-shot text-to-speech (TTS) has been supported.
• [Update Apr. 28, 2024] Recipes for automated audio captioning (AAC) has been supported.
• [Update Mar. 31, 2024] Recipes for automatic speech recognition (ASR) has been supported.

Installation

git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout tags/v4.35.2
pip install -e .
cd ..
git clone https://github.com/huggingface/peft.git
cd peft
git checkout tags/v0.6.0
pip install -e .
cd ..
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
git clone https://github.com/ddlBoJack/SLAM-LLM.git
cd SLAM-LLM
pip install  -e .

For some examples, you may need to use fairseq, the command line is as follows:

# you need to install fairseq before SLAM-LLM
git clone https://github.com/pytorch/fairseq
cd fairseq
pip install --editable ./

We also provide a docker image for convenience:

# build docker image
docker build -t slam-llm:latest .

# run docker image with gpu
docker run -it --gpus all --name slam --shm-size=256g slam-llm:latest /bin/bash

Usage

List of Recipes

We provide reference implementations of various LLM-based speech, audio, and music tasks:

• Speech Task
  • Automatic Speech Recognition (ASR)
  • Text-to-Speech (TTS)
  • Visual Speech Recognition (VSR)
• Audio Task
  • Automated Audio Captioning (AAC)
  • Spatial Audio Understanding
• Music Task
  • Music Caption (MC)

Configuration Priority

We provide hierarchical configuration inheritance relationships as follows:

command-line (shell file) > Hydra configuration (yaml file) > dataclass configuration (Python file)

Features

• Easily extend to new models and tasks.
• Detailed recipes for training and high-performance checkpoints for inference.
• Mixed precision training which trains faster with less GPU memory on NVIDIA tensor cores.
• Multi-GPU training with data and model parallel, supporting DDP, FSDP and deepspeed (still need to be improved).
• Flexible configuration based on Hydra and dataclass allowing a combination of code, command-line and file based configuration.

Acknowledge

• We borrow code from Llama-Recipes for the training process.
• We borrow code from Fairseq for deepspeed configuration.
• We thank the contributors for providing diverse recipes.

Citation

@article{ma2024embarrassingly,
  title={An Embarrassingly Simple Approach for LLM with Strong ASR Capacity},
  author={Ma, Ziyang and Yang, Guanrou and Yang, Yifan and Gao, Zhifu and Wang, Jiaming and Du, Zhihao and Yu, Fan and Chen, Qian and Zheng, Siqi and Zhang, Shiliang and others},
  journal={arXiv preprint arXiv:2402.08846},
  year={2024}
}