Home

Awesome

🌋 LLaVA-Plus: Large Language and Vision Assistants that Plug and Learn to Use Skills

Learning to Use Tools For Creating Multimodal Agents.

[Project Page] [Arxiv] [Demo] [Data] [Model Zoo]

Note: Some sections of the code are currently being prepared and updated. Please stay tuned.

Release

Code License Data License Usage and License Notices: The data and checkpoint are intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaVA, LLaMA, Vicuna, and GPT-4. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.

Contents

Install

If you are not using Linux, do NOT proceed, see instructions for macOS and Windows from LLaVA.

  1. Clone this repository and navigate to the LLaVA-Plus folder
git clone https://github.com/LLaVA-VL/LLaVA-Plus-Codebase LLaVA-Plus
cd LLaVA-Plus
  1. Install Package
conda create -n llava python=3.10 -y
conda activate llava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
  1. Install additional packages for training cases
pip install -e ".[train]"
pip install flash-attn --no-build-isolation

LLaVA-Plus Weights

We are still preparing the part. Stay tuned!

Please check out our Model Zoo for all public LLaVA-Plus checkpoints, and the instructions on how to use the weights.

Demo

Demo Architecture

llava-plus-arch

To run our demo, you have four steps.

  1. Launch a controller: enable to control different works.
  2. Launch a model worker: core llava-plus model.
  3. Launch tool workers: the tools you want to call.
  4. Launch a gradio web server: a front end page for users.

1. Launch a controller

python -m llava.serve.controller --host 0.0.0.0 --port 20001

2. Launch a model worker

This is the actual worker that performs the inference on the GPU. Each worker is responsible for a single model specified in --model-path.

python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:20001 --port 40000 --worker http://localhost:40000 --model-path <huggingface or local path>

Wait until the process finishes loading the model and you see "Uvicorn running on ...". Now, refresh your Gradio web UI, and you will see the model you just launched in the model list.

If you are using an Apple device with an M1 or M2 chip, you can specify the mps device by using the --device flag: --device mps.

<details> <summary>Multiple works</summary> You can launch as many workers as you want, and compare between different model checkpoints in the same Gradio interface. Please keep the `--controller` the same, and modify the `--port` and `--worker` to a different port number for each worker. ```Shell python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:20001 --port <different from 40000, say 40001> --worker http://localhost:<change accordingly, i.e. 40001> --model-path <ckpt2> ``` </details> <details> <summary>Launch a model worker (Multiple GPUs, when GPU VRAM <= 24GB)</summary>

If the VRAM of your GPU is less than 24GB (e.g., RTX 3090, RTX 4090, etc.), you may try running it with multiple GPUs. Our latest code base will automatically try to use multiple GPUs if you have more than one GPU. You can specify which GPUs to use with CUDA_VISIBLE_DEVICES. Below is an example of running with the first two GPUs.

CUDA_VISIBLE_DEVICES=0,1 python -m llava.serve.model_worker --host 0.0.0.0 --controller http://localhost:20001 --port 40000 --worker http://localhost:40000 --model-path <huggingface or local path>
</details>

3. Launch tool workers

You need to open different tool works, as shown in the figure above, which means you need to prepare codes from other projects.

We provide a detailed guideline for different projects.

4. Launch a gradio web server.

python -m llava.serve.gradio_web_server_llava_plus --controller http://localhost:20001 --model-list-mode reload

You just launched the Gradio web interface. Now, you can open the web interface with the URL printed on the screen.

Train

LLaVA training consists of two stages: (1) feature alignment stage, and (2) visual instruction tuning stage.

Our llava-plus is trained from the llava-stage-1-pre-trained projectors.

<details> <summary>Training cost</summary> LLaVA-Plus is trained on 4/8 A100 GPUs with 80GB memory. To train on fewer GPUs, you can reduce the `per_device_train_batch_size` and increase the `gradient_accumulation_steps` accordingly. Always keep the global batch size the same: `per_device_train_batch_size` x `gradient_accumulation_steps` x `num_gpus`. </details> <details> <summary>Download Vicuna checkpoints (automatically)</summary>

Our base model Vicuna v1.5, which is an instruction-tuned chatbot, will be downloaded automatically when you run our provided training scripts. No action is needed.

</details>

Stage 1: Pretrain (feature alignment)

Download pre-trained projector directly as we did.

Or you may train the projector following the guideline in LLaVA.

Stage 2: Tool Augmented Visual Instruction Tuning

  1. Prepare data

Please download the training data, and download the images from constituting datasets:

  1. Start training!

Training script with DeepSpeed ZeRO-2: training_llava_plus_v0_7b.sh or training_llava_plus_v1.3_7b.sh

<details> <summary>If you do not have enough GPU memory:</summary> - Use LoRA. See LLaVA repo for more details. - Replace `zero2.json` with `zero3.json` or `zero3_offload.json`. </details> <details> <summary>If you are interested in finetuning LLaVA(LLaVA-Plus) model to your own task/data:</summary> please check out [`Finetune_Custom_Data.md`](https://github.com/haotian-liu/LLaVA/blob/main/docs/Finetune_Custom_Data.md)。 </details> <details> <summary>Some explainations of options:</summary> </details>

Evaluation

See LLaVA's Instruction on model evaluations.

Citation

If you find LLaVA useful for your research and applications, please cite using this BibTeX:


@article{liu2023llavaplus,
  title={LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents},
  author={Liu, Shilong and Cheng, Hao and Liu, Haotian and Zhang, Hao and Li, Feng and Ren, Tianhe and Zou, Xueyan and Yang, Jianwei and Su, Hang and Zhu, Jun and Zhang, Lei and Gao, Jianfeng and Li, Chunyuan},
  journal={arXiv:2311.05437},
  year={2023}
}

@misc{liu2023llava,
      title={Visual Instruction Tuning}, 
      author={Liu, Haotian and Li, Chunyuan and Wu, Qingyang and Lee, Yong Jae},
      publisher={arXiv:2304.08485},
      year={2023}
}

Acknowledgement

Related Projects

Included Tools

For future project ideas, please check out: