Awesome

Awesome-LLMs-for-Video-Understanding

🔥🔥🔥 Video Understanding with Large Language Models: A Survey

Yunlong Tang^1,*, Jing Bi^1,*, Siting Xu^2,*, Luchuan Song¹, Susan Liang¹ , Teng Wang^2,3 , Daoan Zhang¹ , Jie An¹ , Jingyang Lin¹ , Rongyi Zhu¹ , Ali Vosoughi¹ , Chao Huang¹ , Zeliang Zhang¹ , Pinxin Liu¹ , Mingqian Feng¹ , Feng Zheng² , Jianguo Zhang² , Ping Luo³ , Jiebo Luo¹, Chenliang Xu^1,†. (*Core Contributors, †Corresponding Authors)

¹University of Rochester, ²Southern University of Science and Technology, ³The University of Hong Kong

Paper | Project Page

📢 News

[07/23/2024]

📢 We've recently updated our survey: “Video Understanding with Large Language Models: A Survey”!

✨ This comprehensive survey covers video understanding techniques powered by large language models (Vid-LLMs), training strategies, relevant tasks, datasets, benchmarks, and evaluation methods, and discusses the applications of Vid-LLMs across various domains.

🚀 What's New in This Update: <br>✅ Updated to include around 100 additional Vid-LLMs and 15 new benchmarks as of June 2024. <br>✅ Introduced a novel taxonomy for Vid-LLMs based on video representation and LLM functionality. <br>✅ Added a Preliminary chapter, reclassifying video understanding tasks from the perspectives of granularity and language involvement, and enhanced the LLM Background section. <br>✅ Added a new Training Strategies chapter, removing adapters as a factor for model classification. <br>✅ All figures and tables have been redesigned.

Multiple minor updates will follow this major update. And the GitHub repository will be gradually updated soon. We welcome your reading and feedback ❤️

<font size=5><center><b> Table of Contents </b> </center></font>

• Awesome-LLMs-for-Video-Understanding
  • 🔥🔥🔥 Video Understanding with Large Language Models: A Survey
  • Why we need Vid-LLMs?
  • 😎 Vid-LLMs: Models
    • 📑 Citation
      • 🗒️ Taxonomy 1
        • 🕹️ Video Analyzer × LLM
          • LMM as Summarizer
          • LLM as Manager
        • 👾 Video Embedder × LLM
          • LLM as Text Decoder
          • LLM as Regressor
          • LLM as Hidden Layer
        • 🧭 (Analyzer + Embedder) × LLM
          • LLM as Manager
          • LMM as Summarizer
          • LLM as Text Decoder
          • LLM as Regressor
          • LLM as Hidden Layer
      • 🗒️ Taxonomy 2
        • 🤖 LLM-based Video Agents
        • 🎥 Vid-LLM Pretraining
        • 👀 Vid-LLM Instruction Tuning
          • Fine-tuning with Connective Adapters
          • Fine-tuning with Insertive Adapters
          • Fine-tuning with Hybrid Adapters
        • 🦾 Hybrid Methods
        • Training-free Methods
  • Tasks, Datasets, and Benchmarks
    • Recognition and Anticipation
    • Captioning and Description
    • Grounding and Retrieval
    • Question Answering
    • Video Instruction Tuning
      • Pretraining Dataset
      • Fine-tuning Dataset
    • Video-based Large Language Models Benchmark
  • Contributing
    • 🌟 Star History
    • ♥️ Contributors

Why we need Vid-LLMs?

😎 Vid-LLMs: Models

📑 Citation

If you find our survey useful for your research, please cite the following paper:

@article{vidllmsurvey,
      title={Video Understanding with Large Language Models: A Survey}, 
      author={Tang, Yunlong and Bi, Jing and Xu, Siting and Song, Luchuan and Liang, Susan and Wang, Teng and Zhang, Daoan and An, Jie and Lin, Jingyang and Zhu, Rongyi and Vosoughi, Ali and Huang, Chao and Zhang, Zeliang and Zheng, Feng and Zhang, Jianguo and Luo, Ping and Luo, Jiebo and Xu, Chenliang},
      journal={arXiv preprint arXiv:2312.17432},
      year={2023},
}

🗒️ Taxonomy 1

🕹️ Video Analyzer × LLM

LLM as Summarizer

Title	Model	Date	Code	Venue
Seeing the Unseen: Visual Metaphor Captioning for Videos	GIT-LLaVA	06/2024	code	arXiv
Zero-shot long-form video understanding through screenplay	MM-Screenplayer	06/2024	project page	CVPR
MoReVQA exploring modular reasoning models for video question answering	MoReVQA	04/2024	project page	CVPR
An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLM	IG-VLM	03/2024	code	arXiv
Language repository for long video understanding	LangRepo	03/2024	code	arXiv
Understanding long videos in one multimodal language model pass	MVU	03/2024	code	arXiv
Video ReCap recursive captioning of hour-long videos	Video ReCap	02/2024	code	CVPR
A Simple LLM Framework for Long-Range Video Question-Answering	LLoVi	12/2023	code	arXiv
Grounding-prompter prompting LLM with multimodal information for temporal sentence grounding in long videos	Grounding-prompter	12/2023	code	arXiv
Learning object state changes in videos an open-world perspective	VIDOSC	12/2023	code	CVPR
AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?	AntGPT	07/2023	code	ICLR
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset	VAST	05/2023	code	NeurIPS
VLog: Video as a Long Document	VLog	04/2023	code	-
Learning Video Representations from Large Language Models	LaViLa	12/2022	code	CVPR

LLM as Manager

Title	Model	Date	Code	Venue
DrVideo: Document Retrieval Based Long Video Understanding	DrVideo	06/2024	code	arXiv
OmAgent a multi-modal agent framework for complex video understanding with task divide-and-conquer	OmAgent	06/2024	code	arXiv
Too Many Frames, not all Useful: Efficient Strategies for Long-Form Video QA	LVNet	06/2024	code	arXiv
VideoTree adaptive tree-based video representation for LLM reasoning on long videos	VideoTree	05/2024	code	arXiv
Harnessing Large Language Models for Training-free Video Anomaly Detection	LAVAD	04/2024	code	CVPR
TraveLER a multi-LMM agent framework for video question-answering	TraveLER	04/2024	code	arXiv
GPTSee enhancing moment retrieval and highlight detection via description-based similarity features	GPTSee	03/2024	code	arXiv
Reframe anything LLM agent for open world video reframing	RAVA	03/2024	code	arXiv
SCHEMA state CHangEs MAtter for procedure planning in instructional videos	SCHEMA	03/2024	code	ICLR
TV-TREES multimodal entailment trees for neuro-symbolic video reasoning	TV-TREES	02/2024	code	arXiv
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding	VideoAgent	03/2024	project page	arXiv
VideoAgent long-form video understanding with large language model as agent	VideoAgent	03/2024	code	arXiv
VURF a general-purpose reasoning and self-refinement framework for video understanding	VURF	03/2024	code	arXiv
Why not use your textbook knowledge-enhanced procedure planning of instructional videos	KEPP	03/2024	code	CVPR
DoraemonGPT toward understanding dynamic scenes with large language models	DoraemonGPT	01/2024	code	arXiv
LifelongMemory: Leveraging LLMs for Answering Queries in Long-form Egocentric Videos	LifelongMemory	12/2023	code	arXiv
Zero-Shot Video Question Answering with Procedural Programs	ProViQ	12/2023	code	arXiv
AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn	AssistGPT	06/2023	code	arXiv
ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System	ChatVideo	04/2023	project page	arXiv
Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions	Video ChatCaptioner	04/2023	code	arXiv
ViperGPT: Visual Inference via Python Execution for Reasoning	ViperGPT	03/2023	code	arXiv
Hawk: Learning to Understand Open-World Video Anomalies	Hawk	05/2024	code	arXiv

👾 Video Embedder × LLM

LLM as Text Decoder

LLM as Regressor

LLM as Hidden Layer

🧭 (Analyzer + Embedder) × LLM

LLM as Manager

LLM as Summarizer

LLM as Regressor

LLM as Text Decoder

LLM as Hidden Layer

Title	Model	Date	Code	Venue
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models	PG-Video-LLaVA	11/2023	code	arXiv

🗒️ Taxonomy 2

🤖 LLM-based Video Agents

Title	Model	Date	Code	Venue
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language	Socratic Models	04/2022	project page	arXiv
Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions	Video ChatCaptioner	04/2023	code	arXiv
VLog: Video as a Long Document	VLog	04/2023	code	-
ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding System	ChatVideo	04/2023	project page	arXiv
MM-VID: Advancing Video Understanding with GPT-4V(ision)	MM-VID	10/2023	-	arXiv
MISAR: A Multimodal Instructional System with Augmented Reality	MISAR	10/2023	project page	ICCV
Grounding-Prompter: Prompting LLM with Multimodal Information for Temporal Sentence Grounding in Long Videos	Grounding-Prompter	12/2023	-	arXiv
NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation	NaVid	02/2024	project page -	RSS
VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding	VideoAgent	03/2024	project page	arXiv
VideoINSTA: Zero-shot Long Video Understanding via Informative Spatial-Temporal Reasoning with LLMs	VideoINSTA	09/2024	code	EMNLP

🎥 Vid-LLM Pretraining

Title	Model	Date	Code	Venue
Learning Video Representations from Large Language Models	LaViLa	12/2022	code	CVPR
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning	Vid2Seq	02/2023	code	CVPR
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset	VAST	05/2023	code	NeurIPS
Merlin:Empowering Multimodal LLMs with Foresight Minds	Merlin	12/2023	-	arXiv

👀 Vid-LLM Instruction Tuning

Fine-tuning with Connective Adapters

Title	Model	Date	Code	Venue
Video-LLaMA: An Instruction-Finetuned Visual Language Model for Video Understanding	Video-LLaMA	06/2023	code	arXiv
VALLEY: Video Assistant with Large Language model Enhanced abilitY	VALLEY	06/2023	code	-
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models	Video-ChatGPT	06/2023	code	arXiv
Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration	Macaw-LLM	06/2023	code	arXiv
LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning	LLMVA-GEBC	06/2023	code	CVPR
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks	mPLUG-video	06/2023	code	arXiv
MovieChat: From Dense Token to Sparse Memory for Long Video Understanding	MovieChat	07/2023	code	arXiv
Large Language Models are Temporal and Causal Reasoners for Video Question Answering	LLaMA-VQA	10/2023	code	EMNLP
Video-LLaVA: Learning United Visual Representation by Alignment Before Projection	Video-LLaVA	11/2023	code	arXiv
Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding	Chat-UniVi	11/2023	code	arXiv
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models	LLaMA-VID	11/2023	code	arXiv
VISTA-LLAMA: Reliable Video Narrator via Equal Distance to Visual Tokens	VISTA-LLAMA	12/2023	-	arXiv
Audio-Visual LLM for Video Understanding	-	12/2023	-	arXiv
AutoAD: Movie Description in Context	AutoAD	06/2023	code	CVPR
AutoAD II: The Sequel - Who, When, and What in Movie Audio Description	AutoAD II	10/2023	-	ICCV
AutoAD III: The Prequel -- Back to the Pixels	AutoAD III	04/2024	-	CVPR
Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models	FAVOR	10/2023	code	arXiv
VideoLLaMA2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs	VideoLLaMA2	06/2024	code	arXiv

Fine-tuning with Insertive Adapters

Title	Model	Date	Code	Venue
Otter: A Multi-Modal Model with In-Context Instruction Tuning	Otter	06/2023	code	arXiv
VideoLLM: Modeling Video Sequence with Large Language Models	VideoLLM	05/2023	code	arXiv

Fine-tuning with Hybrid Adapters

Title	Model	Date	Code	Venue
VTimeLLM: Empower LLM to Grasp Video Moments	VTimeLLM	11/2023	code	arXiv
GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation	GPT4Video	11/2023	-	arXiv

🦾 Hybrid Methods

Title	Model	Date	Code	Venue
VideoChat: Chat-Centric Video Understanding	VideoChat	05/2023	code demo	arXiv
PG-Video-LLaVA: Pixel Grounding Large Video-Language Models	PG-Video-LLaVA	11/2023	code	arXiv
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding	TimeChat	12/2023	code	CVPR
Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding	Video-GroundingDINO	12/2023	code	arXiv
A Video Is Worth 4096 Tokens: Verbalize Videos To Understand Them In Zero Shot	Video4096	05/2023		EMNLP

💎 Training-free Methods

Tasks, Datasets, and Benchmarks

Recognition and Anticipation

Captioning and Description

Grounding and Retrieval

Question Answering

Video Instruction Tuning

Pretraining Dataset

Fine-tuning Dataset

Video-based Large Language Models Benchmark

Title	Date	Code	Venue
LVBench: An Extreme Long Video Understanding Benchmark	06/2024	code	-
Video-Bench: A Comprehensive Benchmark and Toolkit for Evaluating Video-based Large Language Models	11/2023	code	-
Perception Test: A Diagnostic Benchmark for Multimodal Video Models	05/2023	code	NeurIPS 2023, ICCV 2023 Workshop
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks	07/2023	code	-
FETV: A Benchmark for Fine-Grained Evaluation of Open-Domain Text-to-Video Generation	11/2023	code	NeurIPS 2023
MoVQA: A Benchmark of Versatile Question-Answering for Long-Form Movie Understanding	12/2023	code	-
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark	12/2023	code	-
TempCompass: Do Video LLMs Really Understand Videos?	03/2024	code	ACL 2024
Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis	06/2024	code	-
VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models	06/2024	code	-

Contributing

We welcome everyone to contribute to this repository and help improve it. You can submit pull requests to add new papers, projects, and helpful materials, or to correct any errors that you may find. Please make sure that your pull requests follow the "Title|Model|Date|Code|Venue" format. Thank you for your valuable contributions!

🌟 Star History

♥️ Contributors

Our project wouldn't be possible without the contributions of these amazing people! Thank you all for making this project better.

Yunlong Tang @ University of Rochester
Jing Bi @ University of Rochester
Siting Xu @ Southern University of Science and Technology
Luchuan Song @ University of Rochester
Susan Liang @ University of Rochester
Teng Wang @ The University of Hong Kong
Daoan Zhang @ University of Rochester
Jie An @ University of Rochester
Jingyang Lin @ University of Rochester
Rongyi Zhu @ University of Rochester
Ali Vosoughi @ University of Rochester
Chao Huang @ University of Rochester
Zeliang Zhang @ University of Rochester
Pinxin Liu @ University of Rochester
Mingqian Feng @ University of Rochester
Feng Zheng @ Southern University of Science and Technology
Jianguo Zhang @ Southern University of Science and Technology
Ping Luo @ University of Hong Kong
Jiebo Luo @ University of Rochester
Chenliang Xu @ University of Rochester