Awesome

Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao

Jun Song, Shiji Song, Gao Huang, Bo Zheng

<span>[ English | <a href="README_zh.md">中文</a> ]</span>

Abstract

High-resolution Large Multimodel Models (LMM) encounter the challenges of excessive visual tokens and quadratic visual complexity. Current high-resolution LMMs address the quadratic complexity while still generating excessive visual tokens. However, the redundancy in visual tokens is the key problem as it leads to more substantial compute. To mitigate this, we propose ConvLLaVA, which employs ConvNeXt, a hierarchical backbone, as the visual encoder of LMM to replace Vision Transformer (ViT). ConvLLaVA compresses high-resolution images into information-rich visual features, effectively avoiding the generation of excessive visual tokens. To enhance the capabilities of ConvLLaVA, we propose two critical optimizations.

• Since the low-resolution pretrained ConvNeXt underperforms when directly applied on high resolution, we update it to merge the gap.
• Furthermore, since ConvNeXt's original compression ratio is insufficient for much higher resolution inputs, we train a successive stage to further compress the visual tokens, effectively reducing redundancy.

These optimizations enable ConvLLaVA to support inputs of 1536x1536 resolution while generating only 576 visual tokens, accommodating images of arbitrary aspect ratios. Experimental results demonstrate that our method achieves competitive performance with state-of-the-art models on mainstream benchmarks.

Release :loudspeaker:

• 2024/05/25: Checkpoints are released.
• 2024/04/17: Our code is released.

If you are interested in Large Multimodal Models or you have great ideas, please feel free to email with me: Chunjiang Ge.

Usage and License Notices: This project utilizes certain datasets and checkpoints that are subject to their respective original licenses. Users must comply with all terms and conditions of these original licenses, including but not limited to the OpenAI Terms of Use for the dataset and the specific licenses for base language models for checkpoints trained using the dataset (e.g. Llama community license for LLaMA-2 and Vicuna-v1.5). This project does not impose any additional constraints beyond those stipulated in the original licenses. Furthermore, users are reminded to ensure that their use of the dataset and checkpoints is in compliance with all applicable laws and regulations.

Contents

• Abstract
• Release :loudspeaker:
• Contents
• TODO
• Install
• Model Zoo
• Dataset
• Train
• Evaluation
• Citation
• Acknowledgement

TODO

• Add LMMs-eval supports.
• Add VLMEvalKit supports.
• Add xtuner supports.
• Release weights.
• Release inference code.

Install

1. Clone this repository and navigate to ConvLLaVA folder

git clone https://github.com/alibaba/conv-llava
cd conv-llava

1. Install Package

conda create -n convllava python=3.11 -y
conda activate convllava
pip install --upgrade pip  # enable PEP 660 support
pip install -e .

3. Install additional packages for training cases

pip install -e ".[train]"
pip install flash-attn --no-build-isolation

Model Zoo

The performance on mainstream benchmarks are shown below:

Please check out our Model Zoo for all public ConvLLaVA checkpoints, and the instructions of how to use the weights.

Dataset

Data we use is introduded in Data.md.

Train

We use the following hyperparameters for training ConvLLaVA.

The training scripts are in the scripts:

• Projector Initialzation: stage1
• Vision Language Pretraining: stage2
• Instruction Tuning: stage3

Evaluation

We support VLMEVALKIT and lmms-eval to evaluate our model now. See Evaluation.md for more details.

Citation

If you find LLaVA useful for your research and applications, please cite using this BibTeX:

@misc{ge2024convllava,
    title={ConvLLaVA: Hierarchical Backbones as Visual
Encoder for Large Multimodal Models},
    author={Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, Bo Zheng},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
    year={2024}
    eprint={2045.15738},
}

Acknowledgement

• Vicuna: the codebase LLaVA built upon, and our base model Vicuna-13B that has the amazing language capabilities!
• LLaVA: the codebase we built upon.