Awesome

<p align=center>Awesome Diffusion Models In Low-level Vision </p>

Contents

• Latest Works Recommended
• Papers
  • Diffusion Models For Natural Image Restoration<details><summary> General-purpose Image Restoration (IR)</summary>
    • Zero-shot DM-based IR
    • Supervised DM-based IR</details><details><summary> Task-specific Image Restoration (IR)</summary>
      • Super Resolution
      • Inpainting
      • Deblur
      • Dehaze
      • Low-light Enhancement
      • Image Fusion</details>
      • Extended Diffusion Models In Low-level Vision<details><summary> Diffusion Models In Low-level Medical Image Analysis</summary>
        • MRI
        • X-ray-based
        • Multi-modal
        • Other Modalities</details><details> <summary> Diffusion Models In Remote Sensing For Low-level Vision Tasks </summary>
          • Visible-light Remote Sensing Image
          • Hyperspectral Imaging (HSI)
          • Synthetic Aperture Radar (SAR)
          • Multi-modal</details><details><summary> Varied Low-level Vision Tasks In Video Through Diffusion Models</summary>
            • Video Frame Prediction and Interpolation
            • Super Resolution For Video Generation
            • Video Restoration</details>
• Related Surveys Recommended
• Large-scale datasets for model pre-training
• Datasets for low-level vision tasks
• Evaluation metrics
• Reference

<a id="3.">Latest Works Recommended</a>

Diffusion Models in Low-Level Vision: A Survey<br />Chunming He, Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang, Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, Xiu Li<br />TPAMI, minor revision. [Paper]

Reti-Diff: Illumination Degradation Image Restoration with Retinex-based Latent Diffusion Model<br />Chunming He, Chengyu Fang, Yulun Zhang, Kai Li, Longxiang Tang, Chenyu You, Fengyang Xiao, Zhenhua Guo, Xiu Li<br /> arXiv 2023. [Paper] [Code]<br /> March 2024<br />

<a id="Papers">Papers</a>

<a id="1.">1. Diffusion Models For Natural Image Restoration</a>

<a id="1.1">1.1 General-purpose Image Restoration</a>

<a id="1.1.1">1.1.1 Zero-shot DM-based IR</a>

<a id="1.1.2">1.1.2 Supervised DM-based IR</a>

<a id="1.2">1.2 Task-specific Image Restoration</a>

<a id="1.2.1">1.2.1 Super Resolution (SR)</a>

<a id="1.2.2">1.2.2 Inpainting</a>

<a id="1.2.3">1.2.3 Deblur</a>

<a id="1.2.4">1.2.4 Dehaze</a>

<a id="1.2.5">1.2.5 Low-light Enhancement</a>

<a id="1.2.6">1.2.6 Image Fusion</a>

<a id="2.">2. Extended Diffusion Models In Low-level Vision</a>

<a id="2.1">2.1 Diffusion Models In Low-level Medical Image Analysis</a>

<a id="2.1.1">2.1.1 MRI</a>

<a id="2.1.2">2.1.2 X-ray-based</a>

<a id="2.1.3">2.1.3 Multi-modal</a>

<a id="2.1.4">2.1.4 Other Modalities</a>

<a id="2.2">2.2 Diffusion Models In Remote Sensing For Low-level Vision Tasks</a>

<a id="2.2.1">2.2.1 Visible-light Remote Sensing Image</a>

<a id="2.2.2">2.2.2 HSI</a>

<a id="2.2.3">2.2.3 SAR</a>

<a id="2.2.4">2.2.4 Multi-modal</a>

<a id="2.3">2.3 Varied Low-level Vision Tasks In Video Through Diffusion Models</a>

<a id="2.3.1">2.3.1 Video Frame Prediction and Interpolation</a>

<a id="2.3.2">2.3.2 Super Resolution For Video Generation</a>

<a id="2.3.3">2.3.3 Video Restoration</a>

<a id="4.">Related Surveys Recommended</a>

Diffusion Models in Low-Level Vision: A Survey<br /> arXiv 2024. [Paper] <br />Jun. 2024<br />

Taming Diffusion Models for Image Restoration: A Review<br /> arXiv 2024. [Paper] <br />Sept. 2024<br />

Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives<br /> arXiv 2024. [Paper] <br />Apr. 2024<br />

Diffusion Models, Image Super-Resolution And Everything: A Survey<br /> arXiv 2024. [Paper] <br /> Jan. 2024<br />

State of the Art on Diffusion Models for Visual Computing<br /> arXiv 2023. [Paper]<br /> Oct. 2023<br />

Diffusion Models for Image Restoration and Enhancement -- A Comprehensive Survey.<br /> arXiv 2023. [Paper] <br /> Aug. 2023<br />

Survey on Diverse Image Inpainting using Diffusion Models<br /> PCEMS 2023. [Paper]<br /> Jun. 2023<br />

Diffusion Models for Medical Image Analysis: A Comprehensive Survey<br /> Medical Image Analysis 2023. [Paper]<br /> Nov. 2022<br />

Diffusion Models in Vision: A Survey<br /> TPAMI 2023. [Paper]<br /> Sep. 2022<br />

Diffusion Models: A Comprehensive Survey of Methods and Applications<br /> ACM Computing Surveys 2023. [Paper]<br /> Sep. 2022<br />

<a id="7.">Large-scale datasets for model pre-training</a>

Here, we provide a more comprehensive overview of the commonly used Large-scale datasets for model pre-training in low-level vision tasks.

• ImageNet <br /> ImageNet is a large-scale dataset with over 14 million natural images spanning over 21k classes, termed ImageNet21K. ImageNet1k, serving as a subset of ImageNet21K, has 1k classes with about 1k images per class, often resized to practical dimensions of 224 × 224 or 256 × 256.
• CelebA <br /> CelebA has 200k facial images, each annotated with 40 attributes, featuring 10k celebrities. CelebA-HQ is a subset having 30k high-resolution facial images with a resolution of 1024 × 1024. Enhanced with detailed annotations, CelebAMask-HQ is derived from pixel-wise facial component labeling (face parsing).
• LSUN <br /> LSUN includes 10 scene categories and 20 object categories, each having about 1 million labeled images, typically resized with a shorter edge of 256 pixels and compressed to JPEG image quality of 75.
• AFHQ <br /> AFHQ comprises around 5,000 high-quality animal face images with three categories: cat, dog, and wildlife, each with a resolution of 512 × 512. Current approaches frequently opt to train diffusion models on specific categories, such as cats.
• FFHQ <br /> FFHQ comprises 70k high-resolution facial images with diverse distributions. Existing methods based on pre-trained DMs undergo training on FFHQ and evaluation on CelebA-HQ to showcase their generalizability.

<a id="5.">Datasets for low-level vision tasks</a>

Due to space limitations, we provide a summary of commonly used datasets for several classical natural low-level vision tasks here, including their scales, sources, modalities, and remarks. Clicking on the dataset will redirect you to its download link.

<a id="6.">Evaluation metrics</a>

Due to space limitations, we only introduced the evaluation metrics involved in the comparative experiments in the survey. Here, we provide a more comprehensive overview of the commonly used metrics in low-level vision tasks.

<a id="6.1"> Distortion-based metrics</a>

• PSNR (Peak Signal to Noise Ratio) <br /> PSNR quantifies the pixel-wise disparity between a corrupted image and its clean image by computing their mean squared error.
• SSIM (Structural Similarity) <br /> SSIM aims to accommodate human visual perception, assesses the likeness between distorted and clean images across three aspects, including contrast, brightness, and structure.

<a id="6.2"> Inception-based metrics</a>

• LPIPS (Learned Perceptual Image Patch Similarity) <br /> LPIPS is a learning-based metric that leverages the pre-trained AlexNet as a feature extractor and adjusts the linear layer to emulate human perception.
• FID (Fréchet inception distance) <br /> FID assesses the fidelity and diversity of generated images by modeling the feature-level multivariate Gaussian distribution of the extracted features, by computing the Fréchet distance of their reference images.
• KID (Kernel Inception Distance) <br /> KID is similar to FID, which also leverages the extracted features for assessment but employs maximum mean discrepancy with a polynomial kernel to measure the distance, showing greater stability in the zero-shot and few-shot conditions. KID, specifically, demonstrates greater stability even with limited samples compared to FID.
• NIQE (Natural Image Quality Evaluator) <br /> A no-reference metric, evaluates the distance between the natural scene statistics of distorted images and natural images modeled with a multivariate Gaussian model.
• DISTS (Deep Image Structure and Texture Similarity) <br /> DISTS notes that texture and structure similarities between two images can be assessed by their feature means and correlations obtained from VGG and thus utilizes an SSIM-like distance measurement within the feature space to determine texture and structure similarities.
• PI <br /> PI is introduced in the PIRM Challenge on perceptual SR, aiming to evaluate the perceptual quality of super-resolved images. Its definition, PI=0.5((10-Ma)+NIQE), incorporates Ma, a no-reference IQA metric for SR.

<a id="Reference">Reference</a>

Awesome-diffusion-low-level-vision-by-yulunzhang

Awesome-low-level-vision-resources

Awesome-Diffusion-Models-in-Medical-Imaging: Diffusion Models in Medical Imaging

Star History