Home

Awesome

Vision Mamba: A Comprehensive Survey and Taxonomy

Abstract: State Space Model (SSM) is a mathematical model used to describe and analyze the behavior of dynamic systems. This model has witnessed numerous applications in several fields, including control theory, signal processing, economics and machine learning. In the field of deep learning, state space models are used to process sequence data, such as time series analysis, natural language processing (NLP) and video understanding. By mapping sequence data to state space, long-term dependencies in the data can be better captured. In particular, modern SSMs have shown strong representational capabilities in NLP, especially in long sequence modeling, while maintaining linear time complexity. Notably, based on the latest state-space models, Mamba \cite{Mamba} merges time-varying parameters into SSMs and formulates a hardware-aware algorithm for efficient training and inference. Given its impressive efficiency and strong long-range dependency modeling capability, Mamba is expected to become a new AI architecture that may outperform Transformer. Recently, a number of works have attempted to study the potential of Mamba in various fields, such as general vision, multi-modal, medical image analysis and remote sensing image analysis, by extending Mamba from natural language domain to visual domain. To fully understand Mamba in the visual domain, we conduct a comprehensive survey and present a taxonomy study. This survey focuses on Mamba's application to a variety of visual tasks and data types, and discusses its predecessors, recent advances and far-reaching impact on a wide range of domains. Since Mamba is now on an upward trend, please actively notice us if you have new findings, and new progress on Mamba will be included in this survey in a timely manner and updated on the website: (https://github.com/lx6c78/Vision-Mamba-A-Comprehensive-Survey-and-Taxonomy).

:star: We will timely update the latest representaive literatures and their released source code on this page. If you have any questions, please don't hesitate to contact us at any of the following emails: liuxiao@stu.cqu.edu.cn, zhangchenxu@cqu.edu.cn, leizhang@cqu.edu.cn

📢 Update Log

Citation

If you find this repository is useful for you, please cite our paper:

@misc{liu2024vision,
      title={Vision Mamba: A Comprehensive Survey and Taxonomy}, 
      author={Xiao Liu and Chenxu Zhang and Lei Zhang},
      year={2024},
      eprint={2405.04404},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

Contents

Related Survey

Latest vision Mamba paper

We plan to update these papers in subsequent versions of our survey.

General Vision

1 High-level/Mid-level Vision

1.1 Vision Backbone with Mamba

1.2 Video Analysis and Understanding

1.3 Down-stream Visual Applications

2 Low-level Vision

2.1 Image Denoising and Enhancement

2.2 Image Restoration

3 3-D Visual Recognition

3.1 Point Could Analysis

3.2 Hyperspectral Imaging Analysis

4 Visual Data Generation

Multi-Modal

1 Heterologous Stream

1.1 Multi-Modal Understanding

1.2 Multimodal large language models

2 Homologous Stream

Vertical Application

1 Remote Sensing Image

1.1 Remote Sensing Image Processing

1.2 Remote Sensing Image Classification

1.3 Remote Sensing Image Change Detection

1.4 Remote Sensing Image Segmentation

1.5 Remote Sensing Image Fusion

2 Medical Image

2.1 Medical Image Segmentation

2.1.1 Preliminary explorations of U-shaped Mamba
2.1.2 Improvements to the U-shaped Mamba
2.1.3 U-shaped Mamba with other methodologies
2.1.4 Multi-Dimensional Medical Data Segmentation

2.2 Pathological Diagnosis

2.3 Deformable Image Registration

2.4 Medical Image Reconstruction

2.5 Other Medical Tasks

Other Domains

coming soon