Awesome

Best Practice

We strongly recommend using VLMEevalKit for its useful features and ready-to-use LVLM implementations.

MMIU

This repository is the official implementation of MMIU.

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models
Fanqing Meng^*, Jin Wang^*, Chuanhao Li^*, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang^#, Wenqi Shao^#
^* MFQ, WJ and LCH contribute equally.
^# SWQ (shaowenqi@pjlab.org.cn) and ZKP (zhangkaipeng@pjlab.org.cn) are correponding authors.

💡 News

• 2024/08/13: We have released the codes.

• 2024/08/08: We have released the dataset at https://huggingface.co/datasets/FanqingM/MMIU-Benchmark 🔥🔥🔥

• 2024/08/05: The datasets and codes are coming soon! 🔥🔥🔥

• 2024/08/05: The technical report of MMIU is released! And check our project page! 🔥🔥🔥

Introduction

Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind.

Evaluation Results Overview

• The closed-source proprietary model GPT-4o from OpenAI has taken a leading position in MMIU, surpassing other models such as InternVL2-pro, InternVL1.5-chat, Claude3.5-Sonnet, and Gemini1.5 flash. Note that the open-source models InternVL2-pro.

• Some powerful LVLMs like InternVL1.5 and GLM4V whose pre-training data do not contain multi-image content even outperform many multi-image models which undergo multi-image supervised fine-tuning (SFT), indicating the strong capacity in single-image understanding is the foundation of multi-image comprehension.

• By comparing performance at the level of image relationships, we conclude that LVLM excels at understanding semantic content in multi-image scenarios but has weaker performance in comprehending temporal and spatial relationships in multi-image contexts.

• The analysis based on the task map reveals that models perform better on high-level understanding tasks such as video captioning which are in-domain tasks, but struggle with 3D perception tasks such as 3D detection and temporal reasoning tasks such as image ordering which are out-of-domain tasks.

• By task learning difficulty analysis, tasks involving ordering, retrieval and massive images cannot be overfitted by simple SFT, suggesting that additional pre-training data or training techniques should be incorporated for improvement.

🏆 Leaderboard

🚀 Quick Start

Here, we mainly use the VLMEvalKit framework for testing, with some separate tests as well. Specifically, for multi-image models, we include the following models:

transformers == 33.0

• XComposer2
• XComposer2_1.8b
• qwen_base
• idefics_9b_instruct
• qwen_chat
• flamingov2

transformers == 37.0

• deepseek_vl_1.3b
• deepseek_vl_7b

transformers == 40.0

• idefics2_8b

For single-image models, we include the following:

transformers == 33.0

• sharecaptioner
• monkey-chat

transformers == 37.0

• sharegpt4v_7b
• llava_v1.5_7b
• glm-4v-9b

transformers == 40.0

• llava_next_vicuna_7b
• MiniCPM-Llama3-V-2_5

We use the VLMEvalKit framework for testing. You can refer to the code in VLMEvalKit/test_models.py. Additionally, for closed-source models, please replace the following part of the code by following the example here:

response = model.generate(tmp) # tmp = image_paths + [question]

For other open-source models, we have provided reference code for Mantis and InternVL1.5-chat. For LLava-Interleave, please refer to the original repository.

💐 Acknowledgement

We expressed sincerely gratitude for the projects listed following:

• VLMEvalKit provides useful out-of-box tools and implements many adavanced LVLMs. Thanks for their selfless dedication.
• The Team of InternVL for apis.

📧 Contact

If you have any questions, feel free to contact Fanqing Meng with mengfanqing33@gmail.com

🖊️ Citation

If you feel MMIU useful in your project or research, please kindly use the following BibTeX entry to cite our paper. Thanks!

@article{meng2024mmiu,
  title={MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models},
  author={Meng, Fanqing and Wang, Jin and Li, Chuanhao and Lu, Quanfeng and Tian, Hao and Liao, Jiaqi and Zhu, Xizhou and Dai, Jifeng and Qiao, Yu and Luo, Ping and others},
  journal={arXiv preprint arXiv:2408.02718},
  year={2024}
}