Home

Awesome

Conffusion: Confidence Intervals for Diffusion Models

<a href="https://www.vision.huji.ac.il/conffusion" target="_blank">Project</a> | <a href="http://arxiv.org/abs/2211.09795" target="_blank">Paper</a> <br>

Official PyTorch Implementation for the "Conffusion: Confidence Intervals for Diffusion Models" paper.

<strong>Conffusion:</strong> Given a corrupted input image, our method Conffusion, repurposes a pretrained diffusion model to generate lower and upper bounds around each reconstructed pixel. The true pixel value is guaranteed to fall within these bounds with probability p. We present the bounds for inpainting (the context is dimmed for visualization) and super-resolution. Tighter intervals provide more information; we visualize the normalized interval size, darker values are tighter intervals

Conffusion: Confidence Intervals for Diffusion Models<br> Eliahu Horwitz, Yedid Hoshen<br> <a href="http://arxiv.org/abs/2211.09795" target="_blank">http://arxiv.org/abs/2211.09795 </a> <br>

Abstract: Diffusion models have become the go-to method for many generative tasks, particularly for image-to-image generation tasks such as super-resolution and inpainting. Current diffusion-based methods do not provide statistical guarantees regarding the generated results, often preventing their use in high-stakes situations. To bridge this gap, we construct a confidence interval around each generated pixel such that the true value of the pixel is guaranteed to fall within the interval with a probability set by the user. Since diffusion models parametrize the data distribution, a straightforward way of constructing such intervals is by drawing multiple samples and calculating their bounds. However, this method has several drawbacks: i) slow sampling speeds ii) suboptimal bounds iii) requires training a diffusion model per task. To mitigate these shortcomings we propose Conffusion, wherein we fine-tune a pre-trained diffusion model to predict interval bounds in a single forward pass. We show that Conffusion outperforms the baseline method while being three orders of magnitude faster.

Getting Started

Setup

  1. Clone the repo:
git clone https://github.com/eliahuhorwitz/Conffusion.git
cd Conffusion
  1. Create a new environment and install the libraries:
python3 -m venv conffusion_venv
source conffusion_venv/bin/activate
pip install -r requirements.txt

  1. Download the data from the <a href="https://drive.google.com/file/d/10nP41Z-cHmLuHrIOhvxe-IZrP-gnK4vm/view?usp=share_link" target="_blank">here</a> and extract it into a dir named datasets

  2. Run the desired model by following the links below:

N-Conffusion

Super-resolution: N-Conffusion code for super-resolution (also includes DM<sub>SB</sub> and DM<sub>SBA</sub> code)

Inpainting: N-Conffusion code for inpainting (also includes DM<sub>SB</sub> and DM<sub>SBA</sub> code)

G-Conffusion

Super-resolution + Inpainting: G-Conffusion code for super-resolution and inpainting

Citation

If you find this useful for your research, please use the following.

@article{horwitz2022conffusion,
  title={Conffusion: Confidence Intervals for Diffusion Models},
  author={Horwitz, Eliahu and Hoshen, Yedid},
  journal={arXiv preprint arXiv:2211.09795},
  year={2022}
}

Acknowledgments