Awesome

Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data

<a href="https://abuolaim.nowaty.com/">Abdullah Abuolaim1</a>     <a href="https://iie.fing.edu.uy/~mdelbra/">Mauricio Delbracio2</a>     <a href="https://research.google/people/DamienKelly/">Damien Kelly2</a>     <a href="https://www.eecs.yorku.ca/~mbrown/">Michael S. Brown1</a>     <a href="https://www.sites.google.com/view/milanfarhome/">Peyman Milanfar2</a> 1York University         2Google Research <img src="./figures/rdpd.gif" width="100%" alt="RDPD summary"> <img src="./figures/teaser-git.png" width="100%" alt="teaser figure">

Reference github repository for the paper Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data. Abuolaim et al., proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2021 (YouTube presentation). If you use our dataset or code, please cite our paper:

@article{abuolaim2021learning,
  title={Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data},
  author={Abuolaim, Abdullah and Delbracio, Mauricio and Kelly, Damien and Brown, Michael S and Milanfar, Peyman},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
  year={2021}
}

Synthetic Dataset

Prerequisites

The code tested with:
- Python 3.8.3
- Numpy 1.19.1
- Scipy 1.5.2
- Wand 0.6.3
- Imageio 2.9.0
- OpenCV 4.4.0
Despite not tested, the code may work with library versions other than the specified

Installation

Clone with HTTPS this project to your local machine

git clone https://github.com/Abdullah-Abuolaim/recurrent-defocus-deblurring-synth-dual-pixel.git
cd ./recurrent-defocus-deblurring-synth-dual-pixel/synthetic_dp_defocus_blur_code/

Synthetic dual-pixel (DP) views based on defocus blur

Download SYNTHIA-SF dataset or visit SYNTHIA downloads
- SYNTHIA-SF dataset contains six image sequences: SEQ1-SEQ6
- Traning: SEQ1, SEQ2, SEQ4, SEQ5, SEQ6
- Testing: SEQ3
Run the code in synthetic_dp_defocus_blur_code directory to start generating data as follows:
```
python synthetic_dp_defocus_blur.py --data_dir ./SYNTHIA-SF/ --radial_dis True
```
- --data_dir: path to the downloaded SYNTHIA-SF directory
- --radial_dis: to apply radial distortion on the generated DP views
Running above will create the generated dataset dd_dp_dataset_synth_vid in synthetic_dp_defocus_blur_code
- Generate synthetic image sequence for each camera set (i.e., the five camera sets defined in the main paper)
- There will be 30 image sequences generated in total (5 camera sets x 6 image sequences)
The generated dataset is organized based on the following directory structure <img src="./figures/dataset_structure.png" width="100%" alt="synthetic dataset structure">
- $dir_name$_c: directory of the final output combined images
- $dir_name$_l: directory of the corresponding DP left view images
- $dir_name$_r: directory of the corresponding DP right view images
- source: images exhibiting defocus blur
- target: the corresponding all-in-focus images
- seq_n: image sequence number

RDPD Codes and Models

Prerequisites

The code tested with:
- Python 3.8.3
- TensorFlow 2.2.0
- Keras 2.4.3
- Numpy 1.19.1
- Scipy 1.5.2
- Scikit-image 0.16.2
- Scikit-learn 0.23.2
- OpenCV 4.4.0
Despite not tested, the code may work with library versions other than the specified

Installation

Clone with HTTPS this project to your local machine

git clone https://github.com/Abdullah-Abuolaim/recurrent-defocus-deblurring-synth-dual-pixel.git
cd ./recurrent-defocus-deblurring-synth-dual-pixel/rdpd_code/

Testing

All the trained models used in the main paper and supplemental material can be downloaded link
Place the downloaded .hdf5 model inside ModelCheckpoints for testing
Download DP defocus deblurring dataset [1] link, or visit project GitHub link
After downloading and generating datasets, place them in the same directory e.g., dd_dp_datasets
Run main.py in rdpd_code directory as follows:
```
python main.py --op_phase test --test_model RDPD+ --data_dir ./dd_dp_datasets/
```
- --op_phase: operation phase training or testing
- --test_model: test model name
- --data_dir: path to the directory that has both datasets i.e., dd_dp_dataset_canon and dd_dp_dataset_synth_vid
The results of the tested models will be saved in results directory that will be created inside rdpd_code

Recall that you might need to change

Training

Download DP defocus deblurring dataset [1] link, or visit project GitHub link
After downloading and generating datasets, place them in the same directory e.g., dd_dp_datasets
Run main.py in rdpd_code directory as follows:
```
python main.py --op_phase train --ms_edge_loss True --data_dir ./dd_dp_datasets/
```
- --op_phase: operation phase training or testing
- --ms_edge_loss: whether to use our edge loss or not in addition to the typical MSE loss
- --data_dir: path to the directory that has both datasets i.e., dd_dp_dataset_canon and dd_dp_dataset_synth_vid
Other training options
- --ms_edge_loss_weight_x: the weight of our edge loss at the vertical direction
- --ms_edge_loss_weight_y: the weight of our edge loss at the horizontal direction
- --patch_size: training patch size
- --img_mini_b: image mini-batch size
- --vid_mini_b: video mini-batch size
- --num_frames: number of video frames
- --epoch: number of training epochs
- --lr: initial learning rate
- --schedule_lr_rate': learning rate scheduler (after how many epochs to decrease)
- --dropout_rate': dropout rate of the convLSTM unit
The trained model and checkpoints will be saved in ModelCheckpoints after each epoch

Contact

Should you have any question/suggestion, please feel free to reach out:

Abdullah Abuolaim (abuolaim@eecs.yorku.ca)

References

[1] Abdullah Abuolaim and Michael S. Brown. Defocus Deblurring Using Dual-Pixel Data. In ECCV, 2020.

Awesome

Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data

Synthetic Dataset

Prerequisites

Installation

Synthetic dual-pixel (DP) views based on defocus blur

RDPD Codes and Models

Prerequisites

Installation

Testing

Training

Contact

Related Links

References