Awesome

Awesome-Remote-Sensing-Multimodal-Large-Language-Models

🔥🔥🔥 Multimodal Large Language Models for Remote Sensing: A Survey
[Project Page]This Page |

School of Artificial Intelligence, OPtics, and ElectroNics (iOPEN), Northwestern Polytechnical University

✨✨✨ Behold our meticulously curated trove of RS-MLLMs resources!!!

🎉🚀💡 The website will be updated in real-time to track the latest state of RS-MLLMs!!!

📑📚🔍 Feast your eyes on an assortment of model architecture, training pipelines, datasets, comprehensive evaluation benchmarks, intelligent agents for remote sensing, techniques for instruction tuning, and much more.

🌟🔥📢 A collection of remote sensing multimodal large language model papers focusing on the vision-language domain.

<font size=7><div align='center' > :apple: Multimodal Large Language Models for Remote Sensing </div></font>

Please share a <font color='orange'>STAR ⭐</font> if this project does help

📢 Latest Updates

In this repository, we will collect and document researchers and their outstanding work related to remote sensing multimodal large language model (vision-language).

• The list will be continuously updated 🔥🔥
• 📦 coming soon! 🚀
• May-22-2024: The first RS-MLLMs review manuscript has been submitted for review. 🔥🔥

<font size=5><center><b> Table of Contents </b> </center></font>

• Awesome Papers
  • Multimodal Large Language Models for Remote Sensing
  • Intelligent Agents for Remote Sensing
  • Vision-Language Pre-training Models for Remote Sensing
  • Survey Papers for Remote Sensing Vision-Language Tasks
  • Others
• Awesome Datasets
  • Datasets of Pre-Training for Alignment
  • Datasets of Multimodal Instruction Tuning
• Latest Evaluation Benchmarks for Remote Sensing Vision-Language Tasks
  • Remote Sensing Image Captioning and Aerial Video Captioning
  • Remote Sensing Visual Question Answering and Remote Sensing Visual Grounding
  • Remote Sensing Image-Text Retrieval
  • Remote Sensing Scene Classification

Awesome Papers

Multimodal Large Language Models for Remote Sensing

Title	Venue	Date	Code	Note
RS-MoE: Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering <br>Lin, H., Hong, D., Ge, S., Luo, C., Jiang, K., Jin, H., and Wen, C<br>	arXiv	2024-11-03	-	-
<br> GeoLLaVA: Efficient Fine-Tuned Vision-Language Models for Temporal Change Detection in Remote Sensing <br>Elgendy, H., Sharshar, A., Aboeitta, A., Ashraf, Y., and Guizani, M. <br>	arXiv	2024-10-25	Github	-
<br> TEOChat: Large Language and Vision Assistant for Temporal Earth Observation Data <br>J. Irvin, Jeremy Andrew, et al. <br>	arXiv	2024-10-08	Github	-
<br> EarthMarker: A Visual Prompting MLLM for Region-level and Point-level Remote Sensing Imagery Comprehension <br>Zhang, W., Cai, M., Zhang, T., Zhuang, Y., and Mao, X. <br>	arXiv	2024-07-18	Github	-
<br> SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding <br>J. Luo et al. <br>	arXiv	2024-06-14	Github	-
<br> RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery <br>Y. Bazi, L. Bashmal, M. M. Al Rahhal, R. Ricci, and F. Melgani. <br>	Remote Sensing	2024-04-23	Github	-
<br> H2RSVLM: Towards Helpful and Honest Remote Sensing Large Vision Language Model <br> C. Pang, W. Jiang, L. Jiayu, L. Yi, S. Jiaxing, L. Weijia, W. Xingxing, W. Shuai, F. Litong, X. Guisong, H.Conghui. <br>	arXiv	2024-03-29	Github	-
Popeye: A Unified Visual-Language Model for Multi-Source Ship Detection from Remote Sensing Imagery <br>W. Zhang, M. Cai, T. Zhang, G. Lei, Y. Zhuang, and X. Mao.<br>	arXiv	2024-03-06	-	-
Large Language Models for Captioning and Retrieving Remote Sensing Images <br>J. D. Silva, J. Magalhaes, and D. Tuia.<br>	arXiv	2024-02-09	-	-
<br> LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model <br>D. Muhtar, Z. Li, F. Gu, X. Zhang, and P. Xiao. <br>	arXiv	2024-02-04	Github	-
<br> EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain <br>W. Zhang, M. Cai, T. Zhang, Y. Zhuang, and X. Mao. <br>	arXiv	2024-01-30	Github	accepted by IEEE-TGRS
<br> SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model <br>Y. Zhan, Z. Xiong, and Y. Yuan. <br>	arXiv	2024-01-18	Github	Dataset
<br> GeoChat: Grounded Large Vision-Language Model for Remote Sensing <br>K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan. <br>	arXiv	2023-11-24	Github	accepted by CVPR-24
<br> RSGPT: A Remote Sensing Vision Language Model and Benchmark <br>Y. Hu, J. Yuan, and C. Wen. <br>	arXiv	2023-07-28	Github	-

Intelligent Agents for Remote Sensing

Title	Venue	Date	Code	Note
RS-Agent: Automating Remote Sensing Tasks through Intelligent Agents <br>W. Xu, Z. Yu, Y. Wang, J. Wang, and M. Peng.<br>	arXiv	2024-06-11	-	-
GeoLLM-Engine: A Realistic Environment for Building Geospatial Copilots <br>S. Singh, M. Fore, D. Stamoulis, and D. Group.<br>	arXiv	2024-04-23	-	-
Evaluating Tool-Augmented Agents in Remote Sensing Platforms <br>S. Singh, M. Fore, and D. Stamoulis.<br>	arXiv	2024-04-23	-	-
<br> Change-Agent: Towards Interactive Comprehensive Remote Sensing Change Interpretation and Analysis <br>C. Liu, K. Chen, H. Zhang, Z. Qi, Z. Zou, and Z. Shi.<br>	arXiv	2024-04-01	Github	-
<br> Remote Sensing ChatGPT: Solving Remote Sensing Tasks with ChatGPT and Visual Models <br>H. Guo, X. Su, C. Wu, B. Du, L. Zhang, and D. Li.<br>	arXiv	2024-01-17	Github	-
Tree-GPT: Modular Large Language Model Expert System for Forest Remote Sensing Image Understanding and Interactive Analysis <br>S. Du, S. Tang, W. Wang, X. Li, and R. Guo.<br>	arXiv	2023-10-07	-	-

Vision-Language Pre-training Models for Remote Sensing

Title	Venue	Date	Code	Note
<br> RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing <br>Z. Zhang, T. Zhao, Y. Guo, and J. Yin.<br>	arXiv	2024-01-02	Github	accepted by IEEE-TGRS
<br> RemoteCLIP: A Vision Language Foundation Model for Remote Sensing <br>F. Liu, D. Chen, Z. Guan, X. Zhou, J. Zhu, and J. Zhou.<br>	T-GRS	2024-04-18	Github	arXiv
Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment <br>U. Mall, C. P. Phoo, M. K. Liu, C. Vondrick, B. Hariharan, and K. Bala.<br>	ICLR	2024-01-16	Project	arXiv
<br> RS-CLIP: Zero Shot Remote Sensing Scene Classification via Contrastive Vision-Language Supervision <br>X. Li, C. Wen, Y. Hu, and N. Zhou.<br>	JAG	2023-09-18	Github	-
<br> Parameter-Efficient Transfer Learning for Remote Sensing Image–Text Retrieval <br>Y. Yuan, Y. Zhan, and Z. Xiong.<br>	T-GRS	2023-08-28	Github	arXiv

Survey Papers for Remote Sensing Vision-Language Tasks

Title	Venue	Date	Code	Note
<br>Towards Vision-Language Geo-Foundation Model: A Survey <br>Y. Zhou, L. Feng, Y. Ke, X. Jiang, J. Yan, and W. Zhang.<br>	arXiv	2024-06-13	Github	arXiv
Vision-Language Models in Remote Sensing: Current progress and future trends <br>X. Li, C. Wen, Y. Hu, Z. Yuan, and X. X. Zhu.<br>	MGRS	2024-04-22	-	-
Language Integration in Remote Sensing: Tasks, datasets, and future directions <br>L. Bashmal, Y. Bazi, F. Melgani, M. M. Al Rahhal, and M. A. Al Zuair.<br>	MGRS	2023-10-11	-	-
Brain-Inspired Remote Sensing Foundation Models and Open Problems: A Comprehensive Survey <br>L. Jiao et al.<br>	JSTARS	2023-09-18	-	-

Others

Title	Venue	Date	Code	Note
On the Foundations of Earth and Climate Foundation Models <br>X. X. Zhu et al.<br>	arXiv	2024-05-07	Github	-
On the Promises and Challenges of Multimodal Foundation Models for Geographical, Environmental, Agricultural, and Urban Planning Applications <br>C. Tan et al.<br>	arXiv	2023-12-23	-	-
<br> Charting New Territories: Exploring the Geographic and Geospatial Capabilities of Multimodal LLMs <br>J. Roberts, T. Lüddecke, R. Sheikh, K. Han, and S. Albanie. <br>	arXiv	2023-11-24	Github	-
The Potential of Visual ChatGPT for Remote Sensing <br>L. P. Osco, E. L. de Lemos, W. N. Gonçalves, A. P. M. Ramos, and J. Marcato Junior.<br>	Remote Sensing	2023-06-22	-	-

Awesome Datasets

Datasets of Pre-Training for Alignment

Title	Venue	Date	Code	Note
<be> RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models <br>J. Ge, Y. Zheng, K. Guo, and J. Liang.<br>	arXiv	2024-08-27	Github	Link
<be> ChatEarthNet: A Global-Scale, High-Quality Image-Text Dataset for Remote Sensing <br>Z. Yuan, Z. Xiong, L. Mou, and X. X. Zhu.<br>	arXiv	2024-02-17	Github	Link
<br> RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing <br>Z. Zhang, T. Zhao, Y. Guo, and J. Yin.<br>	arXiv	2024-01-02	Github	-
<br> SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing <br>Z. Wang, R. Prabha, T. Huang, J. Wu, and R. Rajagopal.<br>	AAAI	2024-03-24	Github	arXiv

Datasets of Multimodal Instruction Tuning

Latest Evaluation Benchmarks for Remote Sensing Vision-Language Tasks

Remote Sensing Image Captioning and Aerial Video Captioning

Remote Sensing Visual Question Answering and Remote Sensing Visual Grounding

Remote Sensing Image-Text Retrieval

Remote Sensing Scene Classification

🤖 Contact

If you have any questions about this project, please feel free to contact zhanyangnwpu@gmail.com.