Awesome

<img src="img/logo.png" style="vertical-align: -10px;" :height="40px" width="40px"> SongComposer

This repository is the official implementation of SongComposer.

<a href="https://arxiv.org/pdf/2402.17645">SongComposer: A Large Language Model for Lyric and Melody Composition in Song Generation</a> <a href="https://mark12ding.github.io/">Shuangrui Ding*1</a>, <a href="https://scholar.google.com/citations?user=iELd-Q0AAAAJ">Zihan Liu*2,3</a>, <a href="https://scholar.google.com/citations?user=FscToE0AAAAJ">Xiaoyi Dong3</a>, <a href="https://panzhang0212.github.io/">Pan Zhang3</a>, <a href="https://shvdiwnkozbw.github.io/">Rui Qian1</a>, <a href="https://conghui.github.io/">Conghui He3</a>, <a href="http://dahua.site/">Dahua Lin3</a>, <a href="https://myownskyw7.github.io/">Jiaqi Wang&dagger;3</a> 1The Chinese University of Hong Kong, 2Beihang University, 3Shanghai AI Laboratory * Equal Contribution. &dagger;Corresponding authors. <a href="https://arxiv.org/pdf/2402.17645"><img src="https://img.shields.io/badge/arXiv-<color>"></a> <a href="https://github.com/pjlab-songcomposer/songcomposer"><img src="https://img.shields.io/badge/Code-red"></a> <a href="https://pjlab-songcomposer.github.io"><img src="https://img.shields.io/badge/Demo-yellow"></a>

📜 News

🚀 [2023/6/12] We enhanced SongComposer to a whole-song level and the demo page is updated!

🚀 [2023/3/21] The finetune code of SongComposer are publicly available and the weights of SongComposer_pretrain and SongComposer_sft are publicly available on Hugging Face🤗.

🚀 [2023/2/28] The paper and demo page are released!

💡 Highlights

🔥 SongComposer composes melodies and lyrics with symbolic song representations, with the benefit of better token efficiency, precise representation, flexible format, and human-readable output.
🔥 SongCompose-PT, a comprehensive pretraining dataset that includes lyrics, melodies, and paired lyrics and melodies in either Chinese or English, will be released.
🔥 SongComposer outperforms advanced LLMs like GPT-4 in tasks such as lyric-to-melody generation, melody-to-lyric generation, song continuation, and text-to-song creation.

👨‍💻 Todo

Release of SongCompose-PT dataset
Online Demo of SongComposer
Code of SongComposer
Demo of SongComposer

🛠️ Usage

Requirements

python 3.9 and above
pytorch 2.0 and above
CUDA 12.0 and above are recommended (this is for GPU users)

Installation

Before running the code, make sure you have setup the environment and installed the required packages. Make sure you meet the above requirements, and then install the dependent libraries. Please refer to the installation section of finetune scripts.

Quickstart

We provide a simple example to show how to use SongComposer-SFT with 🤗 Transformers.

<details> <summary> 🤗 Transformers </summary>

from transformers import AutoTokenizer, AutoModelForCausalLM
ckpt_path = "Mar2Ding/songcomposer_sft"
tokenizer = AutoTokenizer.from_pretrained(ckpt_path, trust_remote_code=True)
model = AutoModel.from_pretrained(ckpt_path, trust_remote_code=True).cuda().half()
prompt = 'Create a song on brave and sacrificing with a rapid pace.'
model.inference(prompt, tokenizer)

</details>

Finetune

Please refer to our finetune scripts.

Inference

We have provide a notebook (inference.ipynb) for the inference stage.

⭐ Samples

Audio samples, including our SongComposer and other baselines, are available on our Demo Page. The samples span four tasks related to song generation, covering both English and Chinese.

✒️ Citation

If you find our work helpful for your research, please consider giving a star ⭐ and citation 📝

@misc{ding2024songcomposer,
      title={SongComposer: A Large Language Model for Lyric and Melody Composition in Song Generation}, 
      author={Shuangrui Ding and Zihan Liu and Xiaoyi Dong and Pan Zhang and Rui Qian and Conghui He and Dahua Lin and Jiaqi Wang},
      year={2024},
      eprint={2402.17645},
      archivePrefix={arXiv},
      primaryClass={cs.SD}
}