Home

Awesome

<div align="center"> <h1> 【NeurIPS 2024】Dense Connector for MLLMs </h1> <h5 align="center">

<a href='https://arxiv.org/abs/2405.13800'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://huggingface.co/collections/HuanjinYao/denseconnector-66500e173fc8c9f05dc98dea'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-blue'></a> zhihu <a href='https://huggingface.co/spaces/HuanjinYao/DenseConnector-v1.5-8B'><img src='https://img.shields.io/badge/🤗-Open%20In%20Spaces-blue.svg'></a>

Huanjin Yao<sup>1,3*</sup>, Wenhao Wu<sup>2*✉️</sup>, Taojiannan Yang<sup>4</sup>, Yuxin Song<sup>3</sup>, Mengxi Zhang<sup>3</sup>, Haocheng Feng<sup>3</sup>, Yifan Sun<sup>3</sup>,

Zhiheng Li<sup>1</sup>, Wanli Ouyang<sup>5</sup>, Jingdong Wang<sup>3</sup>

<sup>1</sup>Tsinghua University, <sup>2</sup>The University of Sydney, <sup>3</sup>Baidu, <sup>4</sup>AWS AI Labs, <sup>5</sup>CUHK

<sup>*</sup>Equal Contribution, <sup>✉️</sup>Corresponding Author

</h5> </div>

News

Contents

Overview

We introduce the Dense Connector - a simple, effective, and plug-and-play vision-language connector that significantly enhances existing MLLMs by leveraging multi-layer visual features, with minimal additional computational overhead! We hope that this work will provide valuable experience and serve as a basic module for future MLLM development!

<div align=center> <img width="795" alt="image" src="images/teaser.jpg"> </div>

The Dense Connector utilizes multi-layer visual features to enhance visual representation and augment the visual perception capabilities of the Multimodal Large Language Models (MLLMs) which can be easily integrated into the current MLLMs. We provide three instantiation methods of Dense Connector: Sparse Token Integration (STI), Sparse Channel Integration (SCI), and Dense Channel Integration (DCI). The Dense Channel Integration achieves the best results.

<div align=center> <img width="795" alt="image" src="images/main.jpg"> </div>

Installation

Please follow the instructions below to install the required packages.

  1. Clone this repository
git clone https://github.com/HJYao00/DenseConnector.git
cd DenseConnector
  1. Install Package
conda create -n dc python=3.10 -y
conda activate dc
cd DenseConnector
pip install --upgrade pip 
pip install -e .
  1. Install additional packages for training Dense Connector
pip install ninja
pip install flash-attn --no-build-isolation

Dataset Preparation and Training

Please refer to the document for dataset preparation and training.

Evaluation

We evaluate the Dense Connector across 19 diverse benchmarks, including 11 image benchmarks and 8 video benchmarks. The testing procedures for both images and videos can be found here.

Model Zoo

Please visit our Model Zoo to access all publicly available Dense Connector checkpoints. We scale the LLM from 2.7B to 70B, incorporating the latest open-source large language model, Llama3-8B-Instruct & Llama3-70B-Instruct

Dialogue Example

We provide several dialogue examples, with additional results available in the paper.

<div align=center> <img width="750" alt="image" src="images/qualitative_results.jpg"> </div>

Citation

If you find this repository is useful, please consider star🌟 this repo and cite🖇️ our paper.

@article{yao2024dense,
  title={Dense Connector for MLLMs},
  author={Yao, Huanjin and Wu, Wenhao and Yang, Taojiannan and Song, YuXin and Zhang, Mengxi and Feng, Haocheng and Sun, Yifan and Li, Zhiheng and Ouyang, Wanli and Wang, Jingdong},
  journal={Advances in Neural Information Processing Systems},
  year={2024}
}

Acknowledgment

We extend our gratitude to the open-source efforts of LLaVA, Mini-Gemini and FreeVA.