Awesome
<p align="center"> <img src="./figures/framework.png" alt="image" style="width:1000px;"> </p>Purpose: We aim to provide a summary of diffusion model-based image processing, including restoration, enhancement, coding, and quality assessment. More papers will be summarized.
Xin Li, Yulin Ren, Xin Jin, Cuiling Lan, Xingrui Wang, Wenjun Zeng, Xinchao Wang, Zhibo Chen
University of Science and Technology of China (USTC), National University of Singapore (NUS), Microsoft Research Asia (MSRA), Eastern Institute of Technology (EIT)
Brief intro: The survey for diffusion model-based IR has been released.
:bookmark: News!!!
- 2023-09-19: Updated new related works before 15/09/2023 in this GitHub.
- 2023-11-24: Updated new related works before 10/11/2023 in this GitHub.
- 2023-12-25: Updated new related works before 25/12/2023 in this GitHub.
- 2024-01-25: Updated new related works before 25/01/2024 in this GitHub.
- 2024-03-25: Updated new related works before 25/03/2024 in this GitHub.
- 2024-04-25: Updated new related works before 25/04/2024 in this GitHub.
- 2024-06-25: Updated new related works before 25/06/2024 in this GitHub.
- 2024-08-25: Updated new related works before 25/08/2024 in this GitHub.
- 2024-10-25: Updated new related works before 25/10/2024 in this GitHub.
- 2024-12-25: Updated new related works before 25/12/2024 in this GitHub.
📌 About new works. If you want to incorporate your studies (e.g., the link of paper or project) on diffusion model-based image processing in this repository. Welcome to raise an issue or email us. We will incorporate it into this repository and our survey report as soon as possible.
🌟 Features
- Survey for diffusion model-based Image Restoration (Arxiv version is released)
- Summary for diffusion model-based Image/Video Compression
- Summary for diffusion model-based Quality Assessment
Diffusion model-based Image Restoration/Enhancement
Table of contents
<!-- - [Survey paper](#survey-paper) - [Table of contents](#table-of-contents) -->- Diffusion model for Image Super resolution
- Diffusion model for Image Restoration
- Diffusion model for Image Inpainting
- Diffusion model for Image Shadow Removal
- Diffusion model for Image Denoising
- Diffusion model for Image Dehazing
- Diffusion model for Image Deblurring
- Diffusion model for Medical IR
- Diffusion model for Low-Light Enchancement
- Diffusion model for other tasks
- Benchmark Datasets
- Diffusion model for Image/video compression
- Diffusion model for Image/video quality assessment <!-- - [Recommended Datasets](#recommended-datasets) - [All Datasets](#all-datasets) -->
Image Super-Resolution
Image Restoration
Image Inpainting
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
RePaint | RePaint: Inpainting using denoising diffusion probabilistic models | Andreas Lugmayr | Zero-Shot | CVPR2022 | Image Inpainting | |
CoPaint | Towards coherent image inpainting using denoising diffusion implicit models | Guanhua Zhang | Zero-Shot | PrePrint'23 | Image Inpainting | |
PVA | Personalized Face Inpainting with Diffusion Models by Parallel Visual Attention | Jianjin Xu | Zero-Shot | PrePrint'23 | Image Inpainting |
Image Shadow Removal
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
BCDiff | Boundary-Aware Divide and Conquer: A Diffusion-based Solution for Unsupervised Shadow Removal | Lanqing Guo | Unsupervised | ICCV 2023 | Image Shadow Removal | |
ShadowDiffusion | Shadowdiffusion: When degradation prior meets diffusion model for shadow removal | Lanqing Guo | Supervised | CVPR2023 | Image Shadow Removal | |
DeS3 | DeS3: Attention-driven Self and Soft Shadow Removal using ViT Similarity and Color Convergence | Yeying Jin | Supervised | AAAI'24 | Image Shadow Removal | |
Diff-Shadow | Diff-Shadow: Global-guided Diffusion Model for Shadow Removal | Jinting Luo | Supervised | Preprint'24 | Image Shadow Removal | |
LFG-Diffusion | Latent Feature-Guided Diffusion Models for Shadow Removal | Kangfu Mei | Supervised | WACV 2024 | Image Shadow Removal | |
Deshadow-Anything | Deshadow-Anything: When Segment Anything Model Meets Zero-Shot shadow removal | Xiao Feng Zhang | Supervised | Preprint'23 | Image Shadow Removal |
Image Denoising
Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|
Diffusion model for generative image denoising | Yutong Xie | Supervised | Preprint'23 | Image Denoising | |
Score-Based Diffusion Models as Principled Priors for Inverse Imaging | Berthy T. Feng | Zero-Shot | ICCV 2023 | Image Denoising, Image Deblurring | |
Real-World Denoising via Diffusion Model | Cheng Yang | Supervised | PrePrint'23 | Image Denoising |
Image Dehazing
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
HazeDDPM | High-quality Image Dehazing with Diffusion Model | Hu Yu | Supervised | Preprint'23 | Image Dehazing | |
-- | Frequency Compensated Diffusion Model for Real-scene Dehazing | Jing Wang | Supervised | Preprint'23 | Image Dehazing |
Image Deblurring
Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|
Deblurring via stochastic refinement | Jay Whang | Supervised | CVPR2022 | Image Deblurring | |
Score-Based Diffusion Models as Principled Priors for Inverse Imaging | Berthy T. Feng | Zero-Shot | ICCV 2023 | Image Denoising, Image Deblurring | |
BlindDPS:Parallel diffusion models of operator and image for blind inverse problems | Hyungjin Chung | Zero-Shot | CVPR2023 | Blind Deblurring | |
HI-Diff:Hierarchical Integration Diffusion Model for Realistic Image Deblurring | Zheng Chen | Supervised | Prepreint'23 | Image Deblurring | |
Swintormer:Efficient Image Deblurring Networks based on Diffusion Models | Kang Chen | Supervised | Prepreint'24 | Image Deblurring |
Medical Restoration (MRI,CT)
Model | Paper | First Author | Training Way | Venue | Topic | Project |
---|---|---|---|---|---|---|
MCG | Improving diffusion models for inverse problems using manifold constraints | Hyungjin Chung | Zero-Shot | NeurIPS 2022 | CT Reconstruction | |
-- | Solving inverse problems in medical imaging with score-based generative models | Yang Song | Zero-Shot | ICLR 2022 | CT Reconstruction | |
AdaDiff | Adaptive diffusion priors for accelerated mri reconstruction | Alper Güngör | Supervised | Preprint'23 | MRI Reconstruction | |
HFS-SDE | High-Frequency Space Diffusion Models for Accelerated MRI | Chentao Cao | Supervised | Preprint'22 | MRI Reconstruction | |
SWORD | Stage-by-stage Wavelet Optimization Refinement Diffusion Model for Sparse-View CT Reconstruction | Kai Xu | Supervised | Preprint'23 | MRI Reconstruction | |
FDB | Learning Fourier-Constrained Diffusion Bridges for MRI Reconstruction | Muhammad U. Mirza | Supervised | Preprint'23 | MRI Reconstruction |
Low-Light Enchancement
Other tasks
Benchmark Datasets
Dataset | Task | Usage | Year |
---|---|---|---|
DIV2K | Image Super-resolution | Training,Testing | 2017 |
Flickr2K | Image Super-resolution | Training | 2017 |
Set5 | Image Super-resolution | Testing | 2012 |
Set14 | Image Super-resolution | Testing | 2012 |
BSD100 | Image Super-resolution | Testing | 2012 |
Manga109 | Image Super-resolution | Testing | 2015 |
Urban100 | Image Super-resolution | Testing | 2015 |
OST300 | Image Super-resolution | Testing | 2018 |
DIV8K | Image Super-resolution | Training,Testing | 2019 |
RealSR | Image Super-resolution | Training,Testing | 2019 |
DRealSR | Image Super-resolution | Training,Testing | 2020 |
GoPro | Image Deblurring | Training,Testing | 2017 |
HIDE | Image Deblurring | Training,Testing | 2019 |
RealBlur | Image Deblurring | Training,Testing | 2020 |
Kodak | Image Denoising | Testing | 1999 |
CBSD68 | Image Denoising | Testing | 2001 |
McMaster | Image Denoising | Testing | 2011 |
ImageNet | Image Classification | Training,Testing | 2010 |
ImageNet1k | Image Classification | Testing | 2020 |
LSUN | Image Classification | Training,Testing | 2015 |
Places365 | Image Classification | Training,Testing | 2019 |
LFW | Face Generation | Training | 2008 |
FFHQ | Face Generation | Training | 2019 |
Celeba-HQ | Face Generation | Training | 2018 |
AFHQ | Face Generation | Training | 2020 |
CelebA | Face Generation | Training | 2015 |
ISTD | Image Shadow Removal | Training,Testing | 2018 |
SRD | Image Shadow Removal | Training,Testing | 2017 |
CSD | Image Desnowing | Training,Testing | 2021 |
Snow100k | Image Desnowing | Training,Testing | 2017 |
SRRS | Image Desnowing | Training,Testing | 2020 |
RainDrop | Image Deraining | Training | 2018 |
RainDropClarity | Image Deraining | Training,Testing | 2024 |
Outdoor-Rain | Image Deraining | Training,Testing | 2019 |
DDN-data | Image Deraining | Training,Testing | 2017 |
SPA-data | Image Deraining | Training,Testing | 2019 |
Rain100H | Image Deraining | Training,Testing | 2017 |
Rain100L | Image Deraining | Training,Testing | 2017 |
Haze-4K | Image Dehazing | Training | 2021 |
Dense-Haze | Image Dehazing | Training | 2019 |
RESIDE | Image Dehazing | Training | 2019 |
Diffusion model-based Image/Video Compression
<a id="compression"></a>
Diffusion model-based Image/Video quality assessment
<a id="IQA"></a>
Model | Paper | First Author | Venue | Topic | Project |
---|---|---|---|---|---|
DifFIQA | DifFIQA: Face Image Quality Assessment Using Denoising Diffusion Probabilistic Models | Žiga Babnik | Preprint'23 | Image quality assessment | |
PFD-IQA | Feature Denoising Diffusion Model for Blind Image Quality Assessment | Xudong Li | Preprint'24 | Image quality assessment | |
DP-IQA | DP-IQA: Utilizing Diffusion Prior for Blind Image Quality Assessment in the Wild | Honghao Fu | Preprint'24 | Image quality assessment | |
GenzIQA | GenzIQA: Generalized Image Quality Assessment using Prompt-Guided Latent Diffusion Models | Diptanu De | Preprint'24 | Image quality assessment | |
DiffV<sup>2</sup>IQA</span> | Diffusion Model Based Visual Compensation Guidance and Visual Difference Analysis for No-Reference Image Quality Assessment | Zhaoyang Wang | Preprint'24 | Image quality assessment |
Cite US
If this work is helpful to you, we expect you can cite this work and star this repo. Thanks.
@article{li2023diffusion,
title={Diffusion Models for Image Restoration and Enhancement--A Comprehensive Survey},
author={Li, Xin and Ren, Yulin and Jin, Xin and Lan, Cuiling and Wang, Xingrui and Zeng, Wenjun and Wang, Xinchao and Chen, Zhibo},
journal={arXiv preprint arXiv:2308.09388},
year={2023}
}
<p align="center">
<a href="https://star-history.com/#lixinustc/Awesome-diffusion-model-for-image-processing&Date">
<img src="https://api.star-history.com/svg?repos=lixinustc/Awesome-diffusion-model-for-image-processing&type=Date" alt="Star History Chart">
</a>
</p>