Home

Awesome

GEM - Generalizing Event-Based Motion Deblurring in Real-World Scenarios

Paper | Supp | Video

Event-based motion deblurring has shown promising results by exploiting low-latency events. However, current approaches are limited in their practical usage, as they assume the same spatial resolution of inputs and specific blurriness distributions. This work addresses these limitations and aims to generalize the performance of event-based deblurring in real-world scenarios. We propose a scale-aware network that allows flexible input spatial scales and enables learning from different temporal scales of motion blur. A two-stage self-supervised learning scheme is then developed to fit real-world data distribution. By utilizing the relativity of blurriness, our approach efficiently ensures the restored brightness and structure of latent images and further generalizes deblurring performance to handle varying spatial and temporal scales of motion blur in a self-distillation manner. Our method is extensively evaluated, demonstrating remarkable performance, and we also introduce a real-world dataset consisting of multi-scale blurry frames and events to facilitate research in event-based deblurring.

<div align=center> <img src="figs/overview.jpg" width="800"> </div>

Environment setup

You can create a new Anaconda environment as follows. <br>

conda create -n gem python=3.7
conda activate gem

Clone this repository.

git clone git@github.com:XiangZ-0/GEM.git

Install the above dependencies and Deformable Convolution V2

cd GEM
pip install -r requirements.txt
cd codes/model/DCN_v2/
sh make.sh

Download model and data

Pretrained models and datasets can be downloaded via One Drive. <br> In our paper, we conduct experiments on three types of data:

<div align=center> <img src="figs/camera_setup.jpg" height="200"> </div> <div align=center> MS-RBD capture system </div> <div align=center> <img src="figs/dataset_detail.png" width="600"> </div> <div align=center> Overview of MS-RBD </div> <div align=center> <img src="figs/dataset_examples1.png" width="800"> </div> <div align=center> <img src="figs/dataset_examples2.png" width="800"> </div> <div align=center> Examples of MS-RBD </div>

Easy start

Initialization

cd codes

Test

python main.py --yaml_path configs/evreds_test.yaml
python main.py --yaml_path configs/hsergb_test.yaml
python main.py --yaml_path configs/msrbd_test.yaml

Deblurred results will be saved in ./results/. Note that the script will automatically compute PSNR and SSIM for Ev-REDS and HS-ERGB datasets. Since MS-RBD is a real-world dataset without ground-truth images, we predict the central sharp latent image for qualitative evaluation in real-world scenarios.

For testing on your own datasets, we recommend packing your data in the MS-RBD format and then modifying the following parameters in configs/msrbd_test.yaml according to your needs.

- load_dir:        # change it to your path to load checkpoints
- root_path:       # change it to your dataset directory
- save_path:       # change it to your result directory
- scale_factor:    # change it according to the spatial resolution ratio of images over events in your dataset

Then it is good to go.

Train

python main.py --yaml_path configs/evreds_train.yaml
python main.py --yaml_path configs/hsergb_train.yaml
python main.py --yaml_path configs/msrbd_train.yaml

If you want to train a model on your own dataset (especially real-world datasets), it is recommended to pack your data in the MS-RBD format and then modify configs/msrbd_train.yaml according to your needs for training. :)

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{zhang2023generalizing,
  title={Generalizing Event-Based Motion Deblurring in Real-World Scenarios},
  author={Zhang, Xiang and Yu, Lei and Yang, Wen and Liu, Jianzhuang and Xia, Gui-Song},
  year={2023},
  booktitle={ICCV},
}

Acknowledgement

This code is built based on the Pytorch Lightning template, LIIF, and Deformable Convolution V2.