Awesome

Awesome-Multimodal-Large-Language-Models

Our MLLM works

🔥🔥🔥 A Survey on Multimodal Large Language Models
Project Page [This Page] | Paper

The first comprehensive survey for Multimodal Large Language Models (MLLMs). :sparkles: </div>

Welcome to add WeChat ID (wmd_ustc) to join our MLLM communication group! :star2: </div>

🔥🔥🔥 VITA: Towards Open-Source Interactive Omni Multimodal LLM

<font size=7><div align='center' > [🍎 Project Page] [📖 arXiv Paper] [🌼 GitHub] </div></font>

[2024.08.12] We are announcing VITA, the first-ever open-source Multimodal LLM that can process Video, Image, Text, and Audio, and meanwhile has an advanced multimodal interactive experience. 🌟

<b>Omni Multimodal Understanding</b>. VITA demonstrates robust foundational capabilities of multilingual, vision, and audio understanding, as evidenced by its strong performance across a range of both unimodal and multimodal benchmarks. ✨

<b>Non-awakening Interaction</b>. VITA can be activated and respond to user audio questions in the environment without the need for a wake-up word or button. ✨

<b>Audio Interrupt Interaction</b>. VITA is able to simultaneously track and filter external queries in real-time. This allows users to interrupt the model's generation at any time with new questions, and VITA will respond to the new query accordingly. ✨

🔥🔥🔥 Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis

<font size=7><div align='center' > [🍎 Project Page] [📖 arXiv Paper] [📊 Dataset][🏆 Leaderboard] </div></font>

[2024.06.03] We are very proud to launch Video-MME, the first-ever comprehensive evaluation benchmark of MLLMs in Video Analysis! 🌟

It applies to both <b>image MLLMs</b>, i.e., generalizing to multiple images, and <b>video MLLMs</b>. Our leaderboard involes SOTA models like Gemini 1.5 Pro, GPT-4o, GPT-4V, LLaVA-NeXT-Video, InternVL-Chat-V1.5, and Qwen-VL-Max. 🌟

It includes both <b>short- (< 2min)</b>, <b>medium- (4min~15min)</b>, and <b>long-term (30min~60min)</b> videos, ranging from <b>11 seconds to 1 hour</b>. ✨

<b>All data are newly collected and annotated by humans, not from any existing video dataset</b>. ✨

🔥🔥🔥 MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
Project Page [Leaderboards] | Paper | :black_nib: Citation

A comprehensive evaluation benchmark for MLLMs. Now the leaderboards include 50+ advanced models, such as Qwen-VL-Max, Gemini Pro, and GPT-4V. :sparkles:

If you want to add your model in our leaderboards, please feel free to email bradyfu24@gmail.com. We will update the leaderboards in time. :sparkles:

The benchmark dataset is collected by Xiamen University for academic research only. You can email yongdongluo@stu.xmu.edu.cn to obtain the dataset, according to the following requirement.

Requirement: A real-name system is encouraged for better academic communication. Your email suffix needs to match your affiliation, such as xx@stu.xmu.edu.cn and Xiamen University. Otherwise, you need to explain why. Please include the information bellow when sending your application email.

Name: (tell us who you are.)
Affiliation: (the name/url of your university or company)
Job Title: (e.g., professor, PhD, and researcher)
Email: (your email address)
How to use: (only for non-commercial use)

<br> 📑 If you find our projects helpful to your research, please consider citing: <br>

@article{fu2023mme,
  title={MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models},
  author={Fu, Chaoyou and Chen, Peixian and Shen, Yunhang and Qin, Yulei and Zhang, Mengdan and Lin, Xu and Yang, Jinrui and Zheng, Xiawu and Li, Ke and Sun, Xing and others},
  journal={arXiv preprint arXiv:2306.13394},
  year={2023}
}

@article{fu2024vita,
  title={VITA: Towards Open-Source Interactive Omni Multimodal LLM},
  author={Fu, Chaoyou and Lin, Haojia and Long, Zuwei and Shen, Yunhang and Zhao, Meng and Zhang, Yifan and Wang, Xiong and Yin, Di and Ma, Long and Zheng, Xiawu and He, Ran and Ji, Rongrong and Wu, Yunsheng and Shan, Caifeng and Sun, Xing},
  journal={arXiv preprint arXiv:2408.05211},
  year={2024}
}

@article{fu2024video,
  title={Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis},
  author={Fu, Chaoyou and Dai, Yuhan and Luo, Yondong and Li, Lei and Ren, Shuhuai and Zhang, Renrui and Wang, Zihan and Zhou, Chenyu and Shen, Yunhang and Zhang, Mengdan and others},
  journal={arXiv preprint arXiv:2405.21075},
  year={2024}
}

@article{yin2023survey,
  title={A survey on multimodal large language models},
  author={Yin, Shukang and Fu, Chaoyou and Zhao, Sirui and Li, Ke and Sun, Xing and Xu, Tong and Chen, Enhong},
  journal={arXiv preprint arXiv:2306.13549},
  year={2023}
}

<font size=5><center><b> Table of Contents </b> </center></font>

• Awesome Papers
  • Multimodal Instruction Tuning
  • Multimodal Hallucination
  • Multimodal In-Context Learning
  • Multimodal Chain-of-Thought
  • LLM-Aided Visual Reasoning
  • Foundation Models
  • Evaluation
  • Multimodal RLHF
  • Others
• Awesome Datasets
  • Datasets of Pre-Training for Alignment
  • Datasets of Multimodal Instruction Tuning
  • Datasets of In-Context Learning
  • Datasets of Multimodal Chain-of-Thought
  • Datasets of Multimodal RLHF
  • Benchmarks for Evaluation
  • Others

Awesome Papers

Multimodal Instruction Tuning

Title	Venue	Date	Code	Demo
<br> mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models <br>	arXiv	2024-08-09	Github	-
<br> VITA: Towards Open-Source Interactive Omni Multimodal LLM <br>	arXiv	2024-08-09	Github	-
<br> LLaVA-OneVision: Easy Visual Task Transfer <br>	arXiv	2024-08-06	Github	Demo
<br> MiniCPM-V: A GPT-4V Level MLLM on Your Phone <br>	arXiv	2024-08-03	Github	Demo
VILA^2: VILA Augmented VILA	arXiv	2024-07-24	-	-
SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Models	arXiv	2024-07-22	-	-
EVLM: An Efficient Vision-Language Model for Visual Understanding	arXiv	2024-07-19	-	-
<br> InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output <br>	arXiv	2024-07-03	Github	Demo
<br> OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding <br>	arXiv	2024-06-27	Github	Local Demo
<br> Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs <br>	arXiv	2024-06-24	Github	Local Demo
<br> Long Context Transfer from Language to Vision <br>	arXiv	2024-06-24	Github	Local Demo
<br> Unveiling Encoder-Free Vision-Language Models <br>	arXiv	2024-06-17	Github	Local Demo
<br> Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models <br>	arXiv	2024-06-12	Github	-
<br> VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs <br>	arXiv	2024-06-11	Github	Local Demo
<br> Parrot: Multilingual Visual Instruction Tuning <br>	arXiv	2024-06-04	Github	-
<br> Ovis: Structural Embedding Alignment for Multimodal Large Language Model <br>	arXiv	2024-05-31	Github	-
<br> Matryoshka Query Transformer for Large Vision-Language Models <br>	arXiv	2024-05-29	Github	Demo
<br> ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models <br>	arXiv	2024-05-24	Github	-
<br> Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models <br>	arXiv	2024-05-24	Github	Demo
<br> Libra: Building Decoupled Vision System on Large Language Models <br>	ICML	2024-05-16	Github	Local Demo
<br> CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts <br>	arXiv	2024-05-09	Github	Local Demo
<br> How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites <br>	arXiv	2024-04-25	Github	Demo
<br> Graphic Design with Large Multimodal Model <br>	arXiv	2024-04-22	Github	-
<br> InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD <br>	arXiv	2024-04-09	Github	Demo
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs	arXiv	2024-04-08	-	-
<br> MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding <br>	CVPR	2024-04-08	Github	-
TOMGPT: Reliable Text-Only Training Approach for Cost-Effective Multi-modal Large Language Model	ACM TKDD	2024-03-28	-	-
<br> Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models <br>	arXiv	2024-03-27	Github	Demo
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training	arXiv	2024-03-14	-	-
<br> MoAI: Mixture of All Intelligence for Large Language and Vision Models <br>	arXiv	2024-03-12	Github	Local Demo
<br> TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document <br>	arXiv	2024-03-07	Github	Demo
<br> The All-Seeing Project V2: Towards General Relation Comprehension of the Open World	arXiv	2024-02-29	Github	-
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation	CVPR	2024-02-26	Coming soon	Coming soon
<br> AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling <br>	arXiv	2024-02-19	Github	-
<br> Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning <br>	arXiv	2024-02-18	Github	-
<br> ALLaVA: Harnessing GPT4V-synthesized Data for A Lite Vision-Language Model <br>	arXiv	2024-02-18	Github	Demo
<br> CoLLaVO: Crayon Large Language and Vision mOdel <br>	arXiv	2024-02-17	Github	-
<br> CogCoM: Train Large Vision-Language Models Diving into Details through Chain of Manipulations <br>	arXiv	2024-02-06	Github	-
<br> MobileVLM V2: Faster and Stronger Baseline for Vision Language Model <br>	arXiv	2024-02-06	Github	-
Enhancing Multimodal Large Language Models with Vision Detection Models: An Empirical Study	arXiv	2024-01-31	Coming soon	-
<br> LLaVA-NeXT: Improved reasoning, OCR, and world knowledge	Blog	2024-01-30	Github	Demo
<br> MoE-LLaVA: Mixture of Experts for Large Vision-Language Models <br>	arXiv	2024-01-29	Github	Demo
<br> InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model <br>	arXiv	2024-01-29	Github	Demo
<br> Yi-VL <br>	-	2024-01-23	Github	Local Demo
SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities	arXiv	2024-01-22	-	-
<br> MobileVLM : A Fast, Reproducible and Strong Vision Language Assistant for Mobile Devices <br>	arXiv	2023-12-28	Github	-
<br> InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks <br>	CVPR	2023-12-21	Github	Demo
<br> Osprey: Pixel Understanding with Visual Instruction Tuning <br>	CVPR	2023-12-15	Github	Demo
<br> CogAgent: A Visual Language Model for GUI Agents <br>	arXiv	2023-12-14	Github	Coming soon
Pixel Aligned Language Models	arXiv	2023-12-14	Coming soon	-
See, Say, and Segment: Teaching LMMs to Overcome False Premises	arXiv	2023-12-13	Coming soon	-
<br> Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models <br>	arXiv	2023-12-11	Github	Demo
<br> Honeybee: Locality-enhanced Projector for Multimodal LLM <br>	arXiv	2023-12-11	Github	-
Gemini: A Family of Highly Capable Multimodal Models	Google	2023-12-06	-	-
<br> OneLLM: One Framework to Align All Modalities with Language <br>	arXiv	2023-12-06	Github	Demo
<br> Lenna: Language Enhanced Reasoning Detection Assistant <br>	arXiv	2023-12-05	Github	-
VaQuitA: Enhancing Alignment in LLM-Assisted Video Understanding	arXiv	2023-12-04	-	-
<br> TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding <br>	arXiv	2023-12-04	Github	Local Demo
<br> Making Large Multimodal Models Understand Arbitrary Visual Prompts <br>	CVPR	2023-12-01	Github	Demo
<br> Dolphins: Multimodal Language Model for Driving <br>	arXiv	2023-12-01	Github	-
<br> LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning <br>	arXiv	2023-11-30	Github	Coming soon
<br> VTimeLLM: Empower LLM to Grasp Video Moments <br>	arXiv	2023-11-30	Github	Local Demo
<br> mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large Language Model <br>	arXiv	2023-11-30	Github	-
<br> LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models <br>	arXiv	2023-11-28	Github	Coming soon
<br> LLMGA: Multimodal Large Language Model based Generation Assistant <br>	arXiv	2023-11-27	Github	Demo
<br> ChartLlama: A Multimodal LLM for Chart Understanding and Generation <br>	arXiv	2023-11-27	Github	-
<br> ShareGPT4V: Improving Large Multi-Modal Models with Better Captions <br>	arXiv	2023-11-21	Github	Demo
<br> LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge <br>	arXiv	2023-11-20	Github	-
<br> An Embodied Generalist Agent in 3D World <br>	arXiv	2023-11-18	Github	Demo
<br> Video-LLaVA: Learning United Visual Representation by Alignment Before Projection <br>	arXiv	2023-11-16	Github	Demo
<br> Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding <br>	CVPR	2023-11-14	Github	-
<br> To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning <br>	arXiv	2023-11-13	Github	-
<br> SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models <br>	arXiv	2023-11-13	Github	Demo
<br> Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models <br>	CVPR	2023-11-11	Github	Demo
<br> LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents <br>	arXiv	2023-11-09	Github	Demo
<br> NExT-Chat: An LMM for Chat, Detection and Segmentation <br>	arXiv	2023-11-08	Github	Local Demo
<br> mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration <br>	arXiv	2023-11-07	Github	Demo
<br> OtterHD: A High-Resolution Multi-modality Model <br>	arXiv	2023-11-07	Github	-
CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding	arXiv	2023-11-06	Coming soon	-
<br> GLaMM: Pixel Grounding Large Multimodal Model <br>	CVPR	2023-11-06	Github	Demo
<br> What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning <br>	arXiv	2023-11-02	Github	-
<br> MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning <br>	arXiv	2023-10-14	Github	Local Demo
<br> Ferret: Refer and Ground Anything Anywhere at Any Granularity <br>	arXiv	2023-10-11	Github	-
<br> CogVLM: Visual Expert For Large Language Models <br>	arXiv	2023-10-09	Github	Demo
<br> Improved Baselines with Visual Instruction Tuning <br>	arXiv	2023-10-05	Github	Demo
<br> LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment <br>	ICLR	2023-10-03	Github	Demo
<br> Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs	arXiv	2023-10-01	Github	-
<br> Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants <br>	arXiv	2023-10-01	Github	Local Demo
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model	arXiv	2023-09-27	-	-
<br> InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition <br>	arXiv	2023-09-26	Github	Local Demo
<br> DreamLLM: Synergistic Multimodal Comprehension and Creation <br>	ICLR	2023-09-20	Github	Coming soon
An Empirical Study of Scaling Instruction-Tuned Large Multimodal Models	arXiv	2023-09-18	Coming soon	-
<br> TextBind: Multi-turn Interleaved Multimodal Instruction-following <br>	arXiv	2023-09-14	Github	Demo
<br> NExT-GPT: Any-to-Any Multimodal LLM <br>	arXiv	2023-09-11	Github	Demo
<br> Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics <br>	arXiv	2023-09-13	Github	-
<br> ImageBind-LLM: Multi-modality Instruction Tuning <br>	arXiv	2023-09-07	Github	Demo
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning	arXiv	2023-09-05	-	-
<br> PointLLM: Empowering Large Language Models to Understand Point Clouds <br>	arXiv	2023-08-31	Github	Demo
<br> ✨Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models <br>	arXiv	2023-08-31	Github	Local Demo
<br> MLLM-DataEngine: An Iterative Refinement Approach for MLLM <br>	arXiv	2023-08-25	Github	-
<br> Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models <br>	arXiv	2023-08-25	Github	Demo
<br> Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities <br>	arXiv	2023-08-24	Github	Demo
<br> Large Multilingual Models Pivot Zero-Shot Multimodal Learning across Languages <br>	ICLR	2023-08-23	Github	Demo
<br> StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data <br>	arXiv	2023-08-20	Github	-
<br> BLIVA: A Simple Multimodal LLM for Better Handling of Text-rich Visual Questions <br>	arXiv	2023-08-19	Github	Demo
<br> Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions <br>	arXiv	2023-08-08	Github	-
<br> The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open World <br>	ICLR	2023-08-03	Github	Demo
<br> LISA: Reasoning Segmentation via Large Language Model <br>	arXiv	2023-08-01	Github	Demo
<br> MovieChat: From Dense Token to Sparse Memory for Long Video Understanding <br>	arXiv	2023-07-31	Github	Local Demo
<br> 3D-LLM: Injecting the 3D World into Large Language Models <br>	arXiv	2023-07-24	Github	-
ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning <br>	arXiv	2023-07-18	-	Demo
<br> BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs <br>	arXiv	2023-07-17	Github	Demo
<br> SVIT: Scaling up Visual Instruction Tuning <br>	arXiv	2023-07-09	Github	-
<br> GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest <br>	arXiv	2023-07-07	Github	Demo
<br> What Matters in Training a GPT4-Style Language Model with Multimodal Inputs? <br>	arXiv	2023-07-05	Github	-
<br> mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document Understanding <br>	arXiv	2023-07-04	Github	Demo
<br> Visual Instruction Tuning with Polite Flamingo <br >	arXiv	2023-07-03	Github	Demo
<br> LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding <br>	arXiv	2023-06-29	Github	Demo
<br> Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic <br>	arXiv	2023-06-27	Github	Demo
<br> MotionGPT: Human Motion as a Foreign Language <br>	arXiv	2023-06-26	Github	-
<br> Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration <br>	arXiv	2023-06-15	Github	Coming soon
<br> LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark <br>	arXiv	2023-06-11	Github	Demo
<br> Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models <br>	arXiv	2023-06-08	Github	Demo
<br> MIMIC-IT: Multi-Modal In-Context Instruction Tuning <br>	arXiv	2023-06-08	Github	Demo
M<sup>3</sup>IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning	arXiv	2023-06-07	-	-
<br> Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding <br>	arXiv	2023-06-05	Github	Demo
<br> LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day <br>	arXiv	2023-06-01	Github	-
<br> GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction <br>	arXiv	2023-05-30	Github	Demo
<br> PandaGPT: One Model To Instruction-Follow Them All <br>	arXiv	2023-05-25	Github	Demo
<br> ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst <br>	arXiv	2023-05-25	Github	-
<br> Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models <br>	arXiv	2023-05-24	Github	Local Demo
<br> DetGPT: Detect What You Need via Reasoning <br>	arXiv	2023-05-23	Github	Demo
<br> Pengi: An Audio Language Model for Audio Tasks <br>	NeurIPS	2023-05-19	Github	-
<br> VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks <br>	arXiv	2023-05-18	Github	-
<br> Listen, Think, and Understand <br>	arXiv	2023-05-18	Github	Demo
<br> VisualGLM-6B <br>	-	2023-05-17	Github	Local Demo
<br> PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering <br>	arXiv	2023-05-17	Github	-
<br> InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning <br>	arXiv	2023-05-11	Github	Local Demo
<br> VideoChat: Chat-Centric Video Understanding <br>	arXiv	2023-05-10	Github	Demo
<br> MultiModal-GPT: A Vision and Language Model for Dialogue with Humans <br>	arXiv	2023-05-08	Github	Demo
<br> X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages <br>	arXiv	2023-05-07	Github	-
<br> LMEye: An Interactive Perception Network for Large Language Models <br>	arXiv	2023-05-05	Github	Local Demo
<br> LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model <br>	arXiv	2023-04-28	Github	Demo
<br> mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality <br>	arXiv	2023-04-27	Github	Demo
<br> MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models <br>	arXiv	2023-04-20	Github	-
<br> Visual Instruction Tuning <br>	NeurIPS	2023-04-17	GitHub	Demo
<br> LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention <br>	ICLR	2023-03-28	Github	Demo
<br> MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning <br>	ACL	2022-12-21	Github	-

Multimodal Hallucination

Title	Venue	Date	Code	Demo
Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation	arXiv	2024-08-01	-	-
<br> Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs <br>	ECCV	2024-07-31	Github	-
<br> Evaluating and Analyzing Relationship Hallucinations in LVLMs <br>	ICML	2024-06-24	Github	-
<br> AGLA: Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention <br>	arXiv	2024-06-18	Github	-
CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models	arXiv	2024-06-04	Coming soon	-
VDGD: Mitigating LVLM Hallucinations in Cognitive Prompts by Bridging the Visual Perception Gap	arXiv	2024-05-24	Coming soon	-
Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback	arXiv	2024-04-22	-	-
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding	arXiv	2024-03-27	-	-
<br> What if...?: Counterfactual Inception to Mitigate Hallucination Effects in Large Multimodal Models <br>	arXiv	2024-03-20	Github	-
Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimization	arXiv	2024-03-13	-	-
<br> Debiasing Multimodal Large Language Models <br>	arXiv	2024-03-08	Github	-
<br> HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding <br>	arXiv	2024-03-01	Github	-
IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding	arXiv	2024-02-28	-	-
<br> Less is More: Mitigating Multimodal Hallucination from an EOS Decision Perspective <br>	arXiv	2024-02-22	Github	-
<br> Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models <br>	arXiv	2024-02-18	Github	-
<br> The Instinctive Bias: Spurious Images lead to Hallucination in MLLMs <br>	arXiv	2024-02-06	Github	-
<br> Unified Hallucination Detection for Multimodal Large Language Models <br>	arXiv	2024-02-05	Github	-
A Survey on Hallucination in Large Vision-Language Models	arXiv	2024-02-01	-	-
Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models	arXiv	2024-01-18	-	-
<br> Hallucination Augmented Contrastive Learning for Multimodal Large Language Model <br>	arXiv	2023-12-12	Github	-
<br> MOCHa: Multi-Objective Reinforcement Mitigating Caption Hallucinations <br>	arXiv	2023-12-06	Github	-
<br> Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites <br>	arXiv	2023-12-04	Github	-
<br> RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback <br>	arXiv	2023-12-01	Github	Demo
<br> OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation <br>	CVPR	2023-11-29	Github	-
<br> Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding <br>	CVPR	2023-11-28	Github	-
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization	arXiv	2023-11-28	Github	Comins Soon
Mitigating Hallucination in Visual Language Models with Visual Supervision	arXiv	2023-11-27	-	-
<br> HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data <br>	arXiv	2023-11-22	Github	-
<br> An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation <br>	arXiv	2023-11-13	Github	-
<br> FAITHSCORE: Evaluating Hallucinations in Large Vision-Language Models <br>	arXiv	2023-11-02	Github	-
<br> Woodpecker: Hallucination Correction for Multimodal Large Language Models <br>	arXiv	2023-10-24	Github	Demo
Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models	arXiv	2023-10-09	-	-
<br> HallE-Switch: Rethinking and Controlling Object Existence Hallucinations in Large Vision Language Models for Detailed Caption <br>	arXiv	2023-10-03	Github	-
<br> Analyzing and Mitigating Object Hallucination in Large Vision-Language Models <br>	ICLR	2023-10-01	Github	-
<br> Aligning Large Multimodal Models with Factually Augmented RLHF <br>	arXiv	2023-09-25	Github	Demo
Evaluation and Mitigation of Agnosia in Multimodal Large Language Models	arXiv	2023-09-07	-	-
CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning	arXiv	2023-09-05	-	-
<br> Evaluation and Analysis of Hallucination in Large Vision-Language Models <br>	arXiv	2023-08-29	Github	-
<br> VIGC: Visual Instruction Generation and Correction <br>	arXiv	2023-08-24	Github	Demo
Detecting and Preventing Hallucinations in Large Vision Language Models	arXiv	2023-08-11	-	-
<br> Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning <br>	ICLR	2023-06-26	Github	Demo
<br> Evaluating Object Hallucination in Large Vision-Language Models <br>	EMNLP	2023-05-17	Github	-

Multimodal In-Context Learning

Title	Venue	Date	Code	Demo
Visual In-Context Learning for Large Vision-Language Models	arXiv	2024-02-18	-	-
<br> Can MLLMs Perform Text-to-Image In-Context Learning? <br>	arXiv	2024-02-02	Github	-
<br> Generative Multimodal Models are In-Context Learners <br>	CVPR	2023-12-20	Github	Demo
Hijacking Context in Large Multi-modal Models	arXiv	2023-12-07	-	-
Towards More Unified In-context Visual Understanding	arXiv	2023-12-05	-	-
<br> MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning <br>	arXiv	2023-09-14	Github	Demo
<br> Link-Context Learning for Multimodal LLMs <br>	arXiv	2023-08-15	Github	Demo
<br> OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models <br>	arXiv	2023-08-02	Github	Demo
<br> Med-Flamingo: a Multimodal Medical Few-shot Learner <br>	arXiv	2023-07-27	Github	Local Demo
<br> Generative Pretraining in Multimodality <br>	ICLR	2023-07-11	Github	Demo
AVIS: Autonomous Visual Information Seeking with Large Language Models	arXiv	2023-06-13	-	-
<br> MIMIC-IT: Multi-Modal In-Context Instruction Tuning <br>	arXiv	2023-06-08	Github	Demo
<br> Exploring Diverse In-Context Configurations for Image Captioning <br>	NeurIPS	2023-05-24	Github	-
<br> Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models <br>	arXiv	2023-04-19	Github	Demo
<br> HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace <br>	arXiv	2023-03-30	Github	Demo
<br> MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action <br>	arXiv	2023-03-20	Github	Demo
<br> ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction <br>	ICCV	2023-03-09	Github	-
<br> Prompting Large Language Models with Answer Heuristics for Knowledge-based Visual Question Answering <br>	CVPR	2023-03-03	Github	-
<br> Visual Programming: Compositional visual reasoning without training <br>	CVPR	2022-11-18	Github	Local Demo
<br> An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA <br>	AAAI	2022-06-28	Github	-
<br> Flamingo: a Visual Language Model for Few-Shot Learning <br>	NeurIPS	2022-04-29	Github	Demo
Multimodal Few-Shot Learning with Frozen Language Models	NeurIPS	2021-06-25	-	-

Multimodal Chain-of-Thought

Title	Venue	Date	Code	Demo
<br> Cantor: Inspiring Multimodal Chain-of-Thought of MLLM <br>	arXiv	2024-04-24	Github	Local Demo
<br> Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models <br>	arXiv	2024-03-25	Github	Local Demo
<br> DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models <br>	NeurIPS	2023-10-25	Github	-
<br> Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic <br>	arXiv	2023-06-27	Github	Demo
<br> Explainable Multimodal Emotion Reasoning <br>	arXiv	2023-06-27	Github	-
<br> EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought <br>	arXiv	2023-05-24	Github	-
Let’s Think Frame by Frame: Evaluating Video Chain of Thought with Video Infilling and Prediction	arXiv	2023-05-23	-	-
T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Large Language Model Signals for Science Question Answering	arXiv	2023-05-05	-	-
<br> Caption Anything: Interactive Image Description with Diverse Multimodal Controls <br>	arXiv	2023-05-04	Github	Demo
Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings	arXiv	2023-05-03	Coming soon	-
<br> Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models <br>	arXiv	2023-04-19	Github	Demo
Chain of Thought Prompt Tuning in Vision Language Models	arXiv	2023-04-16	Coming soon	-
<br> MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action <br>	arXiv	2023-03-20	Github	Demo
<br> Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models <br>	arXiv	2023-03-08	Github	Demo
<br> Multimodal Chain-of-Thought Reasoning in Language Models <br>	arXiv	2023-02-02	Github	-
<br> Visual Programming: Compositional visual reasoning without training <br>	CVPR	2022-11-18	Github	Local Demo
<br> Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering <br>	NeurIPS	2022-09-20	Github	-

LLM-Aided Visual Reasoning

Title	Venue	Date	Code	Demo
<br> Beyond Embeddings: The Promise of Visual Table in Multi-Modal Models <br>	arXiv	2024-03-27	Github	-
<br> V∗: Guided Visual Search as a Core Mechanism in Multimodal LLMs <br>	arXiv	2023-12-21	Github	Local Demo
<br> LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing <br>	arXiv	2023-11-01	Github	Demo
MM-VID: Advancing Video Understanding with GPT-4V(vision)	arXiv	2023-10-30	-	-
<br> ControlLLM: Augment Language Models with Tools by Searching on Graphs <br>	arXiv	2023-10-26	Github	-
<br> Woodpecker: Hallucination Correction for Multimodal Large Language Models <br>	arXiv	2023-10-24	Github	Demo
<br> MindAgent: Emergent Gaming Interaction <br>	arXiv	2023-09-18	Github	-
<br> Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language <br>	arXiv	2023-06-28	Github	Demo
Retrieving-to-Answer: Zero-Shot Video Question Answering with Frozen Large Language Models	arXiv	2023-06-15	-	-
<br> AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn <br>	arXiv	2023-06-14	Github	-
AVIS: Autonomous Visual Information Seeking with Large Language Models	arXiv	2023-06-13	-	-
<br> GPT4Tools: Teaching Large Language Model to Use Tools via Self-instruction <br>	arXiv	2023-05-30	Github	Demo
Mindstorms in Natural Language-Based Societies of Mind	arXiv	2023-05-26	-	-
<br> LayoutGPT: Compositional Visual Planning and Generation with Large Language Models <br>	arXiv	2023-05-24	Github	-
<br> IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models <br>	arXiv	2023-05-24	Github	Local Demo
<br> Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation <br>	arXiv	2023-05-10	Github	-
<br> Caption Anything: Interactive Image Description with Diverse Multimodal Controls <br>	arXiv	2023-05-04	Github	Demo
<br> Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models <br>	arXiv	2023-04-19	Github	Demo
<br> HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace <br>	arXiv	2023-03-30	Github	Demo
<br> MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action <br>	arXiv	2023-03-20	Github	Demo
<br> ViperGPT: Visual Inference via Python Execution for Reasoning <br>	arXiv	2023-03-14	Github	Local Demo
<br> ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions <br>	arXiv	2023-03-12	Github	Local Demo
ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction	ICCV	2023-03-09	-	-
<br> Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models <br>	arXiv	2023-03-08	Github	Demo
<br> Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners <br>	CVPR	2023-03-03	Github	-
<br> From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models <br>	CVPR	2022-12-21	Github	Demo
<br> SuS-X: Training-Free Name-Only Transfer of Vision-Language Models <br>	arXiv	2022-11-28	Github	-
<br> PointCLIP V2: Adapting CLIP for Powerful 3D Open-world Learning <br>	CVPR	2022-11-21	Github	-
<br> Visual Programming: Compositional visual reasoning without training <br>	CVPR	2022-11-18	Github	Local Demo
<br> Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language <br>	arXiv	2022-04-01	Github	-

Foundation Models

Title	Venue	Date	Code	Demo
The Llama 3 Herd of Models	arXiv	2024-07-31	-	-
Chameleon: Mixed-Modal Early-Fusion Foundation Models	arXiv	2024-05-16	-	-
Hello GPT-4o	OpenAI	2024-05-13	-	-
The Claude 3 Model Family: Opus, Sonnet, Haiku	Anthropic	2024-03-04	-	-
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context	Google	2024-02-15	-	-
Gemini: A Family of Highly Capable Multimodal Models	Google	2023-12-06	-	-
Fuyu-8B: A Multimodal Architecture for AI Agents	blog	2023-10-17	Huggingface	Demo
<br> Unified Model for Image, Video, Audio and Language Tasks <br>	arXiv	2023-07-30	Github	Demo
PaLI-3 Vision Language Models: Smaller, Faster, Stronger	arXiv	2023-10-13	-	-
GPT-4V(ision) System Card	OpenAI	2023-09-25	-	-
<br> Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization <br>	arXiv	2023-09-09	Github	-
Multimodal Foundation Models: From Specialists to General-Purpose Assistants	arXiv	2023-09-18	-	-
<br> Bootstrapping Vision-Language Learning with Decoupled Language Pre-training <br>	NeurIPS	2023-07-13	Github	-
<br> Generative Pretraining in Multimodality <br>	arXiv	2023-07-11	Github	Demo
<br> Kosmos-2: Grounding Multimodal Large Language Models to the World <br>	arXiv	2023-06-26	Github	Demo
<br> Transfer Visual Prompt Generator across LLMs <br>	arXiv	2023-05-02	Github	Demo
GPT-4 Technical Report	arXiv	2023-03-15	-	-
PaLM-E: An Embodied Multimodal Language Model	arXiv	2023-03-06	-	Demo
<br> Prismer: A Vision-Language Model with An Ensemble of Experts <br>	arXiv	2023-03-04	Github	Demo
<br> Language Is Not All You Need: Aligning Perception with Language Models <br>	arXiv	2023-02-27	Github	-
<br> BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models <br>	arXiv	2023-01-30	Github	Demo
<br> VIMA: General Robot Manipulation with Multimodal Prompts <br>	ICML	2022-10-06	Github	Local Demo
<br> MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge <br>	NeurIPS	2022-06-17	Github	-
<br> Write and Paint: Generative Vision-Language Models are Unified Modal Learners <br>	ICLR	2022-06-15	Github	-
<br> Language Models are General-Purpose Interfaces <br>	arXiv	2022-06-13	Github	-

Evaluation

Title	Venue	Date	Page
<br> MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation <br>	arXiv	2024-06-29	Github
<br> Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs <br>	arXiv	2024-06-28	Github
<br> CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs <br>	arXiv	2024-06-26	Github
<br> ChartMimic: Evaluating LMM's Cross-Modal Reasoning Capability via Chart-to-Code Generation <br>	arXiv	2024-04-15	Github
<br> Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis <br>	arXiv	2024-05-31	Github
<br> Benchmarking Large Multimodal Models against Common Corruptions <br>	NAACL	2024-01-22	Github
<br> Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs <br>	arXiv	2024-01-11	Github
<br> A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise <br>	arXiv	2023-12-19	Github
<br> BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models <br>	arXiv	2023-12-05	Github
<br> How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs <br>	arXiv	2023-11-27	Github
<br> Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs <br>	arXiv	2023-11-24	Github
<br> MLLM-Bench, Evaluating Multi-modal LLMs using GPT-4V <br>	arXiv	2023-11-23	Github
VLM-Eval: A General Evaluation on Video Large Language Models	arXiv	2023-11-20	Coming soon
<br> Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges <br>	arXiv	2023-11-06	Github
<br> On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Driving <br>	arXiv	2023-11-09	Github
Towards Generic Anomaly Detection and Understanding: Large-scale Visual-linguistic Model (GPT-4V) Takes the Lead	arXiv	2023-11-05	-
A Comprehensive Study of GPT-4V's Multimodal Capabilities in Medical Imaging	arXiv	2023-10-31	-
<br> An Early Evaluation of GPT-4V(ision) <br>	arXiv	2023-10-25	Github
<br> Exploring OCR Capabilities of GPT-4V(ision) : A Quantitative and In-depth Evaluation <br>	arXiv	2023-10-25	Github
<br> HallusionBench: You See What You Think? Or You Think What You See? An Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5, and Other Multi-modality Models <br>	CVPR	2023-10-23	Github
<br> MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V, Bard, and Other Large Multimodal Models <br>	ICLR	2023-10-03	Github
<br> Fool Your (Vision and) Language Model With Embarrassingly Simple Permutations <br>	arXiv	2023-10-02	Github
<br> Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning <br>	arXiv	2023-10-01	Github
<br> Can We Edit Multimodal Large Language Models? <br>	arXiv	2023-10-12	Github
<br> REVO-LION: Evaluating and Refining Vision-Language Instruction Tuning Datasets <br>	arXiv	2023-10-10	Github
The Dawn of LMMs: Preliminary Explorations with GPT-4V(vision)	arXiv	2023-09-29	-
<br> TouchStone: Evaluating Vision-Language Models by Language Models <br>	arXiv	2023-08-31	Github
<br> ✨Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models <br>	arXiv	2023-08-31	Github
<br> SciGraphQA: A Large-Scale Synthetic Multi-Turn Question-Answering Dataset for Scientific Graphs <br>	arXiv	2023-08-07	Github
<br> Tiny LVLM-eHub: Early Multimodal Experiments with Bard <br>	arXiv	2023-08-07	Github
<br> MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities <br>	arXiv	2023-08-04	Github
<br> SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension <br>	CVPR	2023-07-30	Github
<br> MMBench: Is Your Multi-modal Model an All-around Player? <br>	arXiv	2023-07-12	Github
<br> MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models <br>	arXiv	2023-06-23	Github
<br> LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models <br>	arXiv	2023-06-15	Github
<br> LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark <br>	arXiv	2023-06-11	Github
<br> M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models <br>	arXiv	2023-06-08	Github
<br> On The Hidden Mystery of OCR in Large Multimodal Models <br>	arXiv	2023-05-13	Github

Multimodal RLHF

Title	Venue	Date	Code	Demo
<br> Silkie: Preference Distillation for Large Visual Language Models <br>	arXiv	2023-12-17	Github	-
<br> RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback <br>	arXiv	2023-12-01	Github	Demo
<br> Aligning Large Multimodal Models with Factually Augmented RLHF <br>	arXiv	2023-09-25	Github	Demo

Others

Title	Venue	Date	Code	Demo
<br> Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models <br>	arXiv	2024-02-03	Github	-
<br> VCoder: Versatile Vision Encoders for Multimodal Large Language Models <br>	arXiv	2023-12-21	Github	Local Demo
<br> Prompt Highlighter: Interactive Control for Multi-Modal LLMs <br>	arXiv	2023-12-07	Github	-
<br> Planting a SEED of Vision in Large Language Model <br>	arXiv	2023-07-16	Github
<br> Can Large Pre-trained Models Help Vision Models on Perception Tasks? <br>	arXiv	2023-06-01	Github	-
<br> Contextual Object Detection with Multimodal Large Language Models <br>	arXiv	2023-05-29	Github	Demo
<br> Generating Images with Multimodal Language Models <br>	arXiv	2023-05-26	Github	-
<br> On Evaluating Adversarial Robustness of Large Vision-Language Models <br>	arXiv	2023-05-26	Github	-
<br> Grounding Language Models to Images for Multimodal Inputs and Outputs <br>	ICML	2023-01-31	Github	Demo

Awesome Datasets

Datasets of Pre-Training for Alignment

Datasets of Multimodal Instruction Tuning

Datasets of In-Context Learning

Datasets of Multimodal Chain-of-Thought

Datasets of Multimodal RLHF

Benchmarks for Evaluation

Others