Awesome
A-Survey-on-Generative-Diffusion-Model
A curated list for diffusion generative models introduced by the paper--A Survey on Generative Diffusion Model
Table of Contents
- 0. Overview
- 1. Methodology Improvement <!-- - [Variational Gap Optimization](#Variational-Gap-Optimization) --> <!-- - [Dimension Deduction](#Dimension-Deduction) -->
- 2 Application
0. Overview
The original idea of the diffusion probabilistic model is to recreate a specific distribution that starts with random noise.
We provided Diffusion Model.pdf, the slide that serves as a vivid explanation for our article. Here, we not only thank for the articles cited in our survey, but also thank the "Tutorial on Denoising Diffusion-based Generative Modeling: Foundations and Applications" provided by NVIDIA tutorial. Besides, there's also two GitHub Repos for summarizing the up-to-date articles, Awesome-Diffusion-Models and What is the Score?. Thank you for your contribution!
<p align="center"> <img src='./figs/idea.png' width="600"> </p>1. Methodology Improvement
<!-- ### Variational Gap Optimization --> <!-- ### Dimension Deduction -->Nowadays, the main concern of the diffusion model is to speed up its speed and reduce the cost of computing. In general cases, it takes thousands of steps for diffusion models to generate a high-quality sample. Mainly focusing on improving sampling speed, many works from different aspects come into reality. Besides, other problems such as variational gap optimization, distribution diversification, and dimension reduction are also attracting extensive research interests. We divide the improved algorithm w.r.t. problems to be solved. For each problem, we present detailed classification of solutions.
<p align="center"> <img src='./figs/method.png' width="1000"> </p>1.1 Speed-up
1.1.1 Training Scheme
Knowledge DIstillation
-
Progressive distillation for fast sampling of diffusion models
-
ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech
-
Knowledge Distillation in Iterative Generative Models for Improved Sampling Speed
Diffusion Scheme Learning
-
Accelerating Diffusion Models via Early Stop of the Diffusion Process
-
Truncated diffusion probabilistic models
- Zheng, Huangjie and He, Pengcheng and Chen, Weizhu and Zhou, Mingyuan. Arxiv 2022 [pdf]
-
Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
-
How Much is Enough? A Study on Diffusion Times in Score-based Generative Models
- Franzese, Giulio and Rossi, Simone and Yang, Lixuan and Finamore, Alessandro and Rossi, Dario and Filippone, Maurizio and Michiardi, Pietro. Arxiv 2022 [pdf]
-
Poisson Flow Generative Models
-
PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
-
Stable Target Field for Reduced Variance Score Estimation in Diffusion Models
Noise Scale Designing
-
Improved denoising diffusion probabilistic models
-
Noise estimation for generative diffusion models
- San-Roman, Robin and Nachmani, Eliya and Wolf, Lior. Arxiv 2021 [pdf]
-
Fast Sampling of Diffusion Models with Exponential Integrator
-
Variational diffusion models
-
Elucidating the Design Space of Diffusion-Based Generative Models
Data Distribution Replace
- Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise
- Structured denoising diffusion models in discrete state-spaces
1.1.2 Training-Free Sampling
Analytical Method
- Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models
Implicit Sampler
-
Denoising Diffusion Implicit Models
-
gDDIM: Generalized denoising diffusion implicit models
-
Maximum Likelihood Training of Implicit Nonlinear Diffusion Models
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-Chul. Arxiv 2022. [pdf]
Differential Equation Solver Sampler
-
Fast Sampling of Diffusion Models with Exponential Integrator
-
Pseudo numerical methods for diffusion models on manifolds
-
DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps
-
Gotta Go Fast When Generating Data with Score-Based Models
Dynamic Programming Adjustment
-
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
- Watson, Daniel and Chan, William and Ho, Jonathan and Norouzi, Mohammad. ICLR 2022. [pdf]
-
Learning to efficiently sample from diffusion probabilistic models
- Watson, Daniel and Ho, Jonathan and Norouzi, Mohammad and Chan, William. Arxiv 2021. [pdf]
1.1.3 Mixed-Modeling
Acceleration Mixture
-
Tackling the generative learning trilemma with denoising diffusion gans
-
Accelerating Diffusion Models via Early Stop of the Diffusion Process
-
Truncated diffusion probabilistic models
-
DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensiona Latents
-
Diffusion normalizing flow
Expressiveness Mixture
-
Maximum Likelihood Training of Implicit Nonlinear Diffusion Models
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-Chul. Arxiv 2022. [pdf]
-
Score-based generative modeling in latent space
-
Maximum likelihood training of score-based diffusion models
-
Maximum Likelihood Training of Parametrized Diffusion Model
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-chul. [pdf]
-
Diffusion Probabilistic Model Made Slim
- Yang, Xingyi and Zhou, Daquan and Feng, Jiashi and Wang, Xinchao. CVPR 2023. [pdf]
1.1.4 Score-Diffusion Unification
-
Score-based generative modeling through stochastic differential equations
-
PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
-
Variational diffusion models
-
gDDIM: Generalized denoising diffusion implicit models
-
Fast Sampling of Diffusion Models with Exponential Integrator
-
Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality
- Watson, Daniel and Chan, William and Ho, Jonathan and Norouzi, Mohammad. ICLR 2022. [pdf]
-
Interpreting diffusion score matching using normalizing flow
- Gong, Wenbo and Li, Yingzhen. Arxiv 2021. [pdf]
-
Simulating Diffusion Bridges with Score Matching
1.2 Distribution Diversification
1.2.1 Continuous Space
-
Maximum Likelihood Training of Implicit Nonlinear Diffusion Models
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-Chul. Arxiv 2022. [pdf]
-
Score-based generative modeling in latent space
-
Maximum likelihood training of score-based diffusion models
-
Maximum Likelihood Training of Parametrized Diffusion Model
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-chul. [pdf]
-
Diffusion probabilistic models for 3d point cloud generation
-
3d shape generation and completion through point-voxel diffusion
-
A Conditional Point Diffusion-Refinement Paradigm for 3D Point Cloud Completion
1.2.2 Discrete Space
-
Argmax flows and multinomial diffusion: Towards non-autoregressive language models
-
Autoregressive diffusion models
-
A Continuous Time Framework for Discrete Denoising Models
-
Structured denoising diffusion models in discrete state-spaces
-
Improved Vector Quantized Diffusion Models
-
Diffusion bridges vector quantized Variational AutoEncoders
-
Vector Quantized Diffusion Model with CodeUnet for Text-to-Sign Pose Sequences Generation
- Xie, Pan and Zhang, Qipeng and Li, Zexian and Tang, Hao and Du, Yao and Hu, Xiaohui. Arxiv 2022. [pdf]
1.2.3 Constrained Space
-
Pseudo numerical methods for diffusion models on manifolds
-
Riemannian score-based generative modeling
-
Riemannian Diffusion Models
- Huang, Chin-Wei and Aghajohari, Milad and Bose, Avishek Joey and Panangaden, Prakash and Courville, Aaron. Arxiv 2022. [pdf]
-
Permutation invariant graph generation via score-based generative modeling
1.3 Likelihood Optimization
1.3.1 Improved ELBO
Score Connection
-
Maximum likelihood training of score-based diffusion models
-
Soft truncation: A universal training technique of score-based diffusion model for high precision score estimation
Redesign
-
Improved Denoising Diffusion Probabilistic Models
-
Fast Sampling of Diffusion Models with Exponential Integrator
-
Variational diffusion models
-
Structured denoising diffusion models in discrete state-spaces
1.3.2 Variational Gap Optimization
-
Score-based generative modeling in latent space
-
Maximum likelihood training of score-based diffusion models
-
Maximum Likelihood Training of Parametrized Diffusion Model
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-chul. [pdf]
1.4 Dimension Reduction
1.4.1 Mixed-Modeling
-
Maximum likelihood training of score-based diffusion models
-
Maximum Likelihood Training of Parametrized Diffusion Model
- Kim, Dongjun and Na, Byeonghu and Kwon, Se Jung and Lee, Dongsoo and Kang, Wanmo and Moon, Il-chul. [pdf]
2. Application
Benefiting from the powerful ability to generate realistic samples, diffusion models have been widely used in various fields such as computer vision, natural language processing, and bioinformatics.
<p align="center"> <img src='./figs/v9app+.png' width="1000"> </p>2.1 Computer Vision
2.1.1 Low-level Vision
-
Palette: Image-to-image diffusion models
-
Conditional image generation with score-based diffusion models
-
Denoising Diffusion Restoration Models
-
Lossy Compression with Gaussian Diffusion
- Theis, Lucas and Salimans, Tim and Hoffman, Matthew D and Mentzer, Fabian. Arxiv 2022. [pdf]
-
Srdiff: Single image super-resolution with diffusion probabilistic models
-
Repaint: Inpainting using denoising diffusion probabilistic models
2.1.2 High-level Vision
-
Score-based generative modeling in latent space
-
Few-Shot Diffusion Models
- Giannone, Giorgio and Nielsen, Didrik and Winther, Ole. Arxiv 2022。 [pdf]
-
CARD: Classification and Regression Diffusion Models
-
Glide: Towards photorealistic image generation and editing with text-guided diffusion models
-
Segdiff: Image segmentation with diffusion probabilistic models
- Amit, Tomer and Nachmani, Eliya and Shaharbany, Tal and Wolf, Lior. Arxiv 2021. [pdf]
-
ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech
-
Diffusion Causal Models for Counterfactual Estimation
2.1.3 3D Vision
-
Diffusion probabilistic models for 3d point cloud generation
-
A conditional point diffusion-refinement paradigm for 3d point cloud completion
-
3D Shape Generation and Completion Through Point-Voxel Diffusion
-
Score-based point cloud denoising
2.1.4 Video Modeling
-
Video Diffusion Models
- Ho, Jonathan and Salimans, Tim and Gritsenko, Alexey A and Chan, William and Norouzi, Mohammad and Fleet, David J. ICLR Workshop 2022. [pdf]
-
Diffusion probabilistic modeling for video generation
-
Flexible diffusion modeling of long videos
-
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation
-
Diffusion models for video prediction and infilling
-
Conditional Image-to-Video Generation with Latent Flow Diffusion Models
2.1.5 Medical Application
-
Score-based diffusion models for accelerated MRI
-
Solving Inverse Problems in Medical Imaging with Score-Based Generative Models
-
MR Image Denoising and Super-Resolution Using Regularized Reverse Diffusion
- Chung, Hyungjin and Lee, Eun Sun and Ye, Jong Chul. Arxiv 2022. [pdf]
-
What is Healthy? Generative Counterfactual Diffusion for Lesion Localization
2.2 Sequential Modeling
2.2.1 Natural Language Processing
-
Diffusion-LM Improves Controllable Text Generation
-
Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
2.2.2 Time Series
-
CSDI: Conditional score-based diffusion models for probabilistic time series imputation
-
Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models
-
Neural Markov Controlled SDE: Stochastic Optimization for Continuous-Time Data
- Park, Sung Woo and Lee, Kyungjae and Kwon, Junseok. ICLR 2021. [pdf]
2.3 Audio
2.3.1 Sound Generation
-
Palette: Image-to-image diffusion models
-
DiffWave: A Versatile Diffusion Model for Audio Synthesis
-
Grad-TTS: A diffusion probabilistic model for text-to-speech
-
Diff-TTS: A Denoising Diffusion Model for Text-to-Speech
- Myeonghun Jeong and Hyeongju Kim and Sung Jun Cheon and Byoung Jin Choi and Nam Soo Kim. Proc. Interspeech 2021. [pdf]
-
Diffusion-Based Voice Conversion with Fast Maximum Likelihood Sampling Scheme
-
Diffsinger: Singing voice synthesis via shallow diffusion mechanism
-
Diffsound: Discrete Diffusion Model for Text-to-sound Generation
2.3.2 Text to Speech
-
ItoTTS and ItoWave: Linear Stochastic Differential Equation Is All You Need For Audio Generation
-
EdiTTS: Score-based Editing for Controllable Text-to-Speech
-
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance
- Kim, Heeseung and Kim, Sungwon and Yoon, Sungroh. ICML 2022. [pdf]
-
Guided-TTS 2: A Diffusion Model for High-quality Adaptive Text-to-Speech with Untranscribed Data
-
Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models
-
SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping
-
BinauralGrad: A Two-Stage Conditional Diffusion Probabilistic Model for Binaural Audio Synthesis
-
ProDiff: Progressive Fast Diffusion Model For High-Quality Text-to-Speech
2.4 AI for Science
2.4.1 Molecular Conformation Generation
-
Equivariant diffusion for molecule generation in 3d
-
GeoDiff: A Geometric Diffusion Model for Molecular Conformation Generation
-
Learning gradient fields for molecular conformation generation
-
Predicting molecular conformation via dynamic graph score matching
- Luo, Shitong and Shi, Chence and Xu, Minkai and Tang, Jian. NIPS 2021. [pdf]
-
Torsional Diffusion for Molecular Conformer Generation
2.4.2 Material Design
-
Crystal Diffusion Variational Autoencoder for Periodic Material Generation
-
Antigen-Specific Antibody Design and Optimization with Diffusion-Based Generative Models
-
Protein Structure and Sequence Generation with Equivariant Denoising Diffusion Probabilistic Models
-
ProteinSGM: Score-based generative modeling for de novo protein design
- Lee, Jin Sub and Kim, Philip M. Arxiv 2022. [pdf]
Contribute
If you would like to help contribute this list, please feel free to contact me or add pull request with the following Markdown format:
- Paper Name.
- Author List. *Conference Year*. [[pdf]](link) [[code]](link)
This is a Github Summary of our Survey. If you find this file useful in your research, please consider citing:
@article{cao2022survey,
title={A Survey on Generative Diffusion Model},
author={Cao, Hanqun and Tan, Cheng and Gao, Zhangyang and Chen, Guangyong and Heng, Pheng-Ann and Li, Stan Z},
journal={arXiv preprint arXiv:2209.02646},
year={2022}
}
Feedback
If you have any issue about this work, please feel free to contact me by email:
- Hanqun Cao: 1155141481@link.cuhk.edu.hk
- Cheng Tan: tancheng@westlake.edu.cn
- Zhangyang Gao: gaozhangyang@westlake.edu.cn