Home

Awesome

LLaMA2-Accessory: An Open-source Toolkit for LLM Development πŸš€

<p align="center"> <img src="docs/logo.png" width="90%"/> <br> </p> <p align="center"> πŸ“– <a href="https://llama2-accessory.readthedocs.io" target="_blank">Document</a> </p> <p align="center"> πŸ€— <a href="https://huggingface.co/Alpha-VLLM/SPHINX" target="_blank">HF Repo</a> β€’ πŸ‘‹ join our <a href="http://imagebind-llm.opengvlab.com/qrcode/" target="_blank">WeChat</a> β€’ πŸš€ <a href="http://imagebind-llm.opengvlab.com/" target="_blank">Demo</a> </p>

πŸš€LLaMA2-Accessory is an open-source toolkit for pretraining, finetuning and deployment of Large Language Models (LLMs) and multimodal LLMs. This repo is mainly inherited from LLaMA-Adapter with more advanced features.🧠

✨Within this toolkit, we present SPHINX, a versatile multimodal large language model (MLLM) that combines a diverse array of training tasks, data domains, and visual embeddings.

News

Features

Setup

:gear: For environment installation, please refer to Environment Setup.

Model Usage

:robot: Instructions for model pretraining, finetuning, inference, and other related topics are all available in the document.

Frequently Asked Questions (FAQ)

:question: Encountering issues or have further questions? Find answers to common inquiries here. We're here to assist you!

Demos

πŸ’‘ Now, our model SPHINX supports generating high-quality bounding boxes and then present masks created by SAM for all objects within an image driven by input prompts. Give it a try here! πŸš€

<img src="./docs/examples/finetune/mm/sphinx_box_0.png" width="90%" />

Core Contributors

Chris Liu, Ziyi Lin, Guian Fang, Jiaming Han, Yijiang Liu, Renrui Zhang, Longtian Qiu, Yichi Zhang, Siyuan Huang

Project Leader

Peng Gao, Wenqi Shao, Shanghang Zhang

Hiring Announcement

πŸ”₯ We are hiring interns, postdocs, and full-time researchers at the General Vision Group, Shanghai AI Lab, with a focus on multi-modality and vision foundation models. If you are interested, please contact gaopengcuhk@gmail.com.

Citation

If you find our code and paper useful, please kindly cite:

@article{zhang2023llamaadapter,
  title = {LLaMA-Adapter: Efficient Finetuning of Language Models with Zero-init Attention},
  author={Zhang, Renrui and Han, Jiaming and Liu, Chris and Gao, Peng and Zhou, Aojun and Hu, Xiangfei and Yan, Shilin and Lu, Pan and Li, Hongsheng and Qiao, Yu},
  journal={arXiv preprint arXiv:2303.16199},
  year={2023}
}
@article{gao2023llamaadapterv2,
  title = {LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model},
  author={Gao, Peng and Han, Jiaming and Zhang, Renrui and Lin, Ziyi and Geng, Shijie and Zhou, Aojun and Zhang, Wei and Lu, Pan and He, Conghui and Yue, Xiangyu and Li, Hongsheng and Qiao, Yu},
  journal={arXiv preprint arXiv:2304.15010},
  year={2023}
}

Acknowledgement

<details><summary>Show More</summary> </details>

License

Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.