Awesome

InstructIR: High-Quality Image Restoration Following Human Instructions (ECCV 2024)

<a href="https://colab.research.google.com/drive/1OrTvS-i6uLM2Y8kIkq8ZZRwEQxQFchfq?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="google colab logo"></a>

Marcos V. Conde, Gregor Geigle, Radu Timofte

Computer Vision Lab, University of Wuerzburg | Sony PlayStation, FTG

<a href="https://mv-lab.github.io/InstructIR/"><img src="images/instructir.gif" alt="InstructIR" width=100%></a>

Video courtesy of Gradio (see their post about InstructIR). Also shoutout to AK -- see his tweet.

TL;DR: quickstart

InstructIR takes as input an image and a human-written instruction for how to improve that image. The neural model performs all-in-one image restoration. InstructIR achieves state-of-the-art results on several restoration tasks including image denoising, deraining, deblurring, dehazing, and (low-light) image enhancement.

🚀 You can start with the demo tutorial

TODO / News 🔥

• Upload Model weights and results for other InstructIR variants (3D, 5D).

• download all the test datasets for all-in-one restoration.

• check the instructions below to run eval_instructir.py and get all the metrics and results for all-in-one restoration.

• You can download all the qualitative results here instructir_results.zip

• Upload models to HF 🤗 (download the models here)

• 🤗 Hugging Face Demo try it now

• Google Colab Tutorial (check demo.ipynb)

Try it / Tutorial

Try it directly on 🤗 Hugging Face at no cost, no code.

🚀 You can start with the demo tutorial. We also host the same tutorial on google colab so you can run it using free GPUs!.

<a href="https://mv-lab.github.io/InstructIR/"><img src="images/instructir_teaser.png" alt="InstructIR" width=100%></a>

Results

Check test.py and eval_instructir.py. The following command provides all the metric for all the benchmarks using the pre-trained models in models/. The results from InstructIR are saved in the indicated folder results/

python eval_instructir.py --model models/im_instructir-7d.pt --lm models/lm_instructir-7d.pt --device 0 --config configs/eval5d.yml --save results/

An example of the output log is:

>>> Eval on CBSD68_15 noise 0
CBSD68_15_base 24.84328738380881
CBSD68_15_psnr 33.98722295200123 68
CBSD68_15_ssim 0.9315137801801457

....

You can download all the test datasets, and locate them in test-data/. Make sure the paths are updated in the config file configs/eval5d.yml.

You can download all the paper results -check releases-. We test InstructIR in the following benchmarks:

In releases or clicking the link above you can download instructir_results.zip which includes all the qualitative results for those datasets [1.9 Gbs].

Control and Interact

Sometimes the blur, rain, or film grain noise are pleasant effects and part of the "aesthetics". Here we show a simple example on how to interact with InstructIR.

As you can see our model accepts diverse humman-written prompts, from ambiguous to precise instructions. How does it work? Imagine we have the following image as input:

Now we can use InstructIR. with the following prompt (1):

I love this photo, could you remove the raindrops? please keep the content intact

Now, let's enhance the image a bit further (2).

Can you make it look stunning? like a professional photo

The final result looks indeed stunning 🤗 You can do it yourself in the demo tutorial.

FAQS

Disclaimer: please remember this is not a product, thus, you will notice some limitations. As most all-in-one restoration methods, it struggles to generalize on real-world images -- we are working on improving it.

• How should I start? Check our demo Tutorial and also our google collab notebook.

• How can I compare with your method? You can download the results for several benchmarks above on Results.

• How can I test the model? I just want to play with it: Visit our 🤗 Hugging Face demo and test ir for free,

• Why aren't you using diffusion-based models? (1) We want to keep the solution simple and efficient. (2) Our priority is high-fidelity --as in many industry scenarios realted to computational photography--.

Gradio Demo <a href='https://github.com/gradio-app/gradio'><img src='https://img.shields.io/github/stars/gradio-app/gradio'></a>

We made a simple Gradio demo you can run (locally) on your machine here. You need Python>=3.9 and these requirements for it: pip install -r requirements_gradio.txt

python app.py

Acknowledgments

This work was partly supported by the The Humboldt Foundation (AvH). Marcos Conde is also supported by Sony Interactive Entertainment, FTG.

This work is inspired in InstructPix2Pix.

Contacts

For any inquiries contact Marcos V. Conde: <a href="mailto:marcos.conde@uni-wuerzburg.de">marcos.conde [at] uni-wuerzburg.de</a>

Citation BibTeX

@inproceedings{conde2024high,
  title={InstructIR: High-Quality Image Restoration Following Human Instructions},
  author={Conde, Marcos V and Geigle, Gregor and Timofte, Radu},
  booktitle    = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year={2024}
}