Awesome

This is the repository of Foundation Models for Remote Sensing and Earth Observation: A Survey, a comprehensive survey of recent progress in multimodal foundation models for remote sensing and earth observation. For details, please refer to:

Foundation Models for Remote Sensing and Earth Observation: A Survey
[Paper]

Abstract

Remote Sensing (RS) is a crucial technology for observing, monitoring, and interpreting our planet, with broad applications across geoscience, economics, humanitarian fields, etc. While artificial intelligence (AI), particularly deep learning, has achieved significant advances in RS, unique challenges persist in developing more intelligent RS systems, including the complexity of Earth's environments, diverse sensor modalities, distinctive feature patterns, varying spatial and spectral resolutions, and temporal dynamics. Meanwhile, recent breakthroughs in large Foundation Models (FMs) have expanded AI’s potential across many domains due to their exceptional generalizability and zero-shot transfer capabilities. However, their success has largely been confined to natural data like images and video, with degraded performance and even failures for RS data of various non-optical modalities. This has inspired growing interest in developing Remote Sensing Foundation Models (RSFMs) to address the complex demands of Earth Observation (EO) tasks, spanning the surface, atmosphere, and oceans. This survey systematically reviews the emerging field of RSFMs. It begins with an outline of their motivation and background, followed by an introduction of their foundational concepts. It then categorizes and reviews existing RSFM studies including their datasets and technical contributions across Visual Foundation Models (VFMs), Visual-Language Models (VLMs), Large Language Models (LLMs), and beyond. In addition, we benchmark these models against publicly available datasets, discuss existing challenges, and propose future research directions in this rapidly evolving field.

Citation

If you find our work useful in your research, please consider citing:

@article{xiao2024foundation,
  title={Foundation Models for Remote Sensing and Earth Observation: A Survey},
  author={Xiao, Aoran and Xuan, Weihao and Wang, Junjue and Huang, Jiaxing and Tao, Dacheng and Lu, Shijian and Yokoya, Naoto},
  journal={arXiv preprint arXiv:2410.16602},
  year={2024}
}

Menu

• Visual Foundation models (VFMs) for RS
  • VFM Datasets
  • VFM Models
    • Pre-training studies
    • SAM-based studies
• Vision-Language Models for RS
  • VLM Datasets
  • VML Models
• Large Language Models for RS
• Generative Foundation Models for RS
• Other RSFMs

Visual Foundation models for RS

VFM Datasets

VFM Models

Pre-training studies

1. An empirical study of remote sensing pretraining. TGRS2022. | paper | code |
2. Satlaspretrain: A large-scale dataset for remote sensing image understanding. ICCV2023. | paper | code |
3. Seasonal contrast: Unsupervised pre-training from uncurated remote sensing data. ICCV2021. | paper | code |
4. Geography-aware self-supervised learning. ICCV2021. | paper | code |
5. Self-supervised material and texture representation learning for remote sensing tasks. CVPR2022. | paper | code |
6. Change-aware sampling and contrastive learning for satellite images. CVPR2023. | paper | code |
7. Csp: Self-supervised contrastive spatial pre-training for geospatial-visual representations. ICML2023. | paper | code |
8. Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. CVPR2024. | paper | code |
9. Contrastive ground-level image and remote sensing pre-training improves representation learning for natural world imagery. ECCV2024. | paper | code |
10. Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. NeurIPS2022. | paper | code |
11. Towards geospatial foundation models via continual pretraining. ICCV2023. | paper | code |
12. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. ICCV2023. | paper | code |
13. Bridging remote sensors with multisensor geospatial foundation models. CVPR2024. | paper | code |
14. Rethinking transformers pre-training for multi-spectral satellite imagery. CVPR2024. | paper | code |
15. Masked angle-aware autoencoder for remote sensing images. ECCV2024. | paper | code |
16. Mmearth: Exploring multi-modal pretext tasks for geospatial representation learning. ECCV2024. | paper | code |
17. Croma: Remote sensing representations with contrastive radar-optical masked autoencoders. NeurIPS2023. | paper | code |
18. Cross-scale mae: A tale of multiscale exploitation in remote sensing. NeurIPS2023. | paper | code |

SAM-based studies

1. SAMRS: Scaling-up Remote Sensing Segmentation Dataset with Segment Anything Model. NeurIPS2023 (DB). | paper | code |
2. Sam-assisted remote sensing imagery semantic segmentation with object and boundary constraints. TGRS2024. | paper | code |
3. Uv-sam: Adapting segment anything model for urban village identification. AAAI2024. | paper | code |
4. Cs-wscdnet: Class activation mapping and segment anything model-based framework for weakly supervised change detection. TGRS2023. | paper | code |
5. Adapting segment anything model for change detection in vhr remote sensing images. TGRS2024. | paper | code |
6. Segment any change. NeurIPS2024. | paper |
7. Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. TGRS2024. | paper | code |
8. Ringmo-sam: A foundation model for segment anything in multimodal remote-sensing images. TGRS2023. | paper
9. The segment anything model (sam) for remote sensing applications: From zero to one shot. JSTAR2023. | paper | code |
10. Cat-sam: Conditional tuning for few-shot adaptation of segmentation anything model. ECCV2024 (oral). | paper | code |
11. Segment anything with multiple modalities. arXiv2024. | paper | code |

Vision-Language Models for RS

VLM Datasets

VLM Models

1. Remoteclip: A vision language foundation model for remote sensing. TGRS2024. | paper | code |
2. Rs5m: A large scale vision-language dataset for remote sensing vision-language foundation model. TGRS2024. | paper | code |
3. Skyscript: A large and semantically diverse vision-language dataset for remote sensing. AAAI2024. | paper | code |
4. Remote sensing vision-language foundation models without annotations via ground remote alignment. ICLR2024. | paper |
5. Csp: Self-supervised contrastive spatial pre-training for geospatial-visual representations. ICML2023. | paper | code |
6. Geoclip: Clip-inspired alignment between locations and images for effective worldwide geo-localization. NeurIPS2024. | paper | code |
7. Satclip: Global, general-purpose location embeddings with satellite imagery. arXiv2023. | paper | code |
8. Learning representations of satellite images from metadata supervision. ECCV2024. | paper |

Large Language Models for RS

1. Geollm: Extracting geospatial knowledge from large language models. ICLR2024. | paper | code |

Generative Foundation Models for RS

1. Diffusionsat: A generative foundation model for satellite imagery. ICLR2024. | paper | code |
2. MMM-RS: A Multi-modal, Multi-GSD, Multi-scene Remote Sensing Dataset and Benchmark for Text-to-Image Generation. NeurIPS2024.

Other RSFMs

weather forecasting

1. Accurate medium-range global weather forecasting with 3d neural networks. Nature, 2023. | paper | code |