Awesome

VmambaIR: Visual State Space Model for Image Restoration

Yuan Shi, Bin Xia, Xiaoyu Jin, Xing Wang, Tianyu Zhao, Xin Xia, Xuefeng Xiao, and Wenming Yang, "VmambaIR: Visual State Space Model for Image Restoration", arXiv, 2024

[arXiv] [supplementary material] [visual results] [pretrained models]

🔥🔥🔥 News

• 2024-03-18: This repo is released.

Abstract: Image restoration is a critical task in low-level computer vision, aiming to restore high-quality images from degraded inputs. Various models, such as convolutional neural networks (CNNs), generative adversarial networks (GANs), transformers, and diffusion models (DMs), have been employed to address this problem with significant impact. However, CNNs have limitations in capturing long-range dependencies. DMs require large prior models and computationally intensive denoising steps. Transformers have powerful modeling capabilities but face challenges due to quadratic complexity with input image size. To address these challenges, we propose VmambaIR, which introduces State Space Models (SSMs) with linear complexity into comprehensive image restoration tasks. We utilize a Unet architecture to stack our proposed Omni Selective Scan (OSS) blocks, consisting of an OSS module and an Efficient Feed-Forward Network (EFFN). Our proposed omni selective scan mechanism overcomes the unidirectional modeling limitation of SSMs by efficiently modeling image information flows in all six directions. Furthermore, we conducted a comprehensive evaluation of our VmambaIR across multiple image restoration tasks, including image deraining, single image super-resolution, and real-world image super-resolution. Extensive experimental results demonstrate that our proposed VmambaIR achieves state-of-the-art (SOTA) performance with much fewer computational resources and parameters. Our research highlights the potential of state space models as promising alternatives to the transformer and CNN architectures in serving as foundational frameworks for next-generation low-level visual tasks.

Single Image Super-Resolution

<img src="figs/Snipaste_2024-03-18_22-28-23.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-22-34.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-25-10.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-26-41.png" height="216"/>

Real-World Image Super-Resolution

<img src="figs/Snipaste_2024-03-18_22-29-05.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-34-00.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-34-25.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-36-36.png" height="216"/>

Image Deraining

<img src="figs/Snipaste_2024-03-18_22-37-57.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-40-59.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-42-11.png" height="216"/> <img src="figs/Snipaste_2024-03-18_22-44-20.png" height="216"/>

⚒️ TODO

• Release code and pretrained models

🔗 Contents

1. Datasets
2. Models
3. Training
4. Testing
5. Results
6. Citation
7. Acknowledgements

<a name="results"></a>🔎 Results

We achieved state-of-the-art performance on multiple image restoration tasks. Detailed results can be found in the paper.

• quantitative comparisons in Table 1 of the main paper

• visual comparison in Figure 5 of the main paper

• quantitative comparisons in Table 2 of the main paper

• visual comparison in Figure 6 of the main paper

• quantitative comparisons in Table 2 of the main paper

• visual comparison in Figure 6 of the main paper

<a name="citation"></a>📎 Citation

If you find the code helpful in your resarch or work, please cite the following paper(s).

@article{shi2024vmambair,
  title={VmambaIR: Visual State Space Model for Image Restoration},
  author={Shi, Yuan and Xia, Bin and Jin, Xiaoyu and Wang, Xing and Zhao, Tianyu and Xia, Xin and Xiao, Xuefeng and Yang, Wenming},
  journal={arXiv preprint arXiv:2403.11423},
  year={2024}
}

<a name="acknowledgements"></a>💡 Acknowledgements

This code is built on BasicSR, Vmamba, Restormer and DiffIR. We thank them all for their excellent work.