Awesome

Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning

Reference github repository for the paper Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning. Abuolaim et al., proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2022 (YouTube presentation). If you use our dataset or code, please cite our paper:

@inproceedings{abuolaim2022improving,
  title={Improving Single-Image Defocus Deblurring: How Dual-Pixel Images Help Through Multi-Task Learning},
  author={Abuolaim, Abdullah and Afifi, Mahmoud and Brown, Michael S},
  booktitle={Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision},
  pages={1231--1239},
  year={2022}
}

DLDP Dataset

We collected a new diverse and large Dual-Pixel (DLDP) dataset of 2353 scenes. This dataset consists of 7059 images i.e., 2353 images with their 4706 dual-pixel (DP) sub-aperture views.

• Dataset

  • 2090 images used for training (processed to an sRGB and encoded with a lossless 16-bit depth).
  • 263 images used for testing (processed to an sRGB and encoded with a lossless 16-bit depth).

• Training and testing sets

  • The dataset is divided randomly into:
    • 89% training and 11% testing.
  • Each set has a balanced number of indoor/outdoor scenes and aperture sizes.
  • The 500 scenes of the DPDD dataset [1] are added to the training set.

Our New Image Motion Attribute (NIMAT) Effect

Code and Trained Models

Prerequisites

• The code tested with:
  • Python 3.8.3
  • TensorFlow 2.2.0
  • Keras 2.4.3
  • Numpy 1.19.1
  • Scipy 1.5.2
  • Scikit-image 0.16.2
  • Scikit-learn 0.23.2
  • OpenCV 4.4.0
  <i>Despite not tested, the code may work with library versions other than the specified</i>

Installation

• Clone with HTTPS this project to your local machine

git clone https://github.com/Abdullah-Abuolaim/multi-task-defocus-deblurring-dual-pixel-nimat.git
cd ./multi-task-defocus-deblurring-dual-pixel-nimat/mdp_code/

Testing

• Download the final trained model of the second phase used in the main paper i.e., mdp_phase_2_wacv

• Place the downloaded .hdf5 model inside ModelCheckpoints for testing

• Download the DPDD dataset [1], or visit project GitHub

• Run main.py in mdp_code directory as follows:

  python main.py   --op_phase test   --path_to_data $PATH_TO_DPDD_DATA$   --test_model  mdp_phase_2_wacv
  

  • <i>--training_phase</i>: training phase i.e., phase_1 or phase_2
  • <i>--op_phase</i>: operation phase training or testing
  • <i>--path_to_data</i>: path to the directory that has the DPDD data e.g., ./dd_dp_dataset_canon/
  • <i>--test_model</i>: test model name
  • <i>--downscale_ratio</i>: downscale input test image in case the GPU memory is not sufficient, or use CPU instead
  • The results of the tested models will be saved in the results directory that will be created inside mdp_code

• The above testing is for evaluating the DPDD dataset quantitively and quantitatively

  • This evaluation is for data that has ground truth
  • This testing is designed based on the directory structure from the DPDD GitHub

<i>Recall that you might need to change </i>

Quick Testing

• For quick qualitative testing of images in a directory, run the following command:

  python main.py   --op_phase test_quick   --path_to_data $PATH_TO_DATA$   --test_model  mdp_phase_2_wacv
  

  • <i>--path_to_data</i>: path to the directory that has the images (no ground truth)
  • <i>--downscale_ratio</i>: downscale input test image in case the GPU memory is not sufficient, or use CPU instead
  • The results of the tested models will be saved in the results directory that will be created inside mdp_code

<i>Recall that you might need to change </i>

Training

• As described in the main paper, we train our multi-task DP network (MDP) in two phases:
  • First phase: freezing the weights of the deblurring <i>Decoder</i>, then, training with image patches from our new DLDP dataset to optimize for the DP-view synthesis task.
  • Second phase: unfreezing the weights of the deblurring <i>Decoder</i>, then, fine-tuning using images from the DPDD dataset [1] to optimize jointly for both the defocus deblurring and DP-view synthesis tasks.

Phase 1 Training

• Run main.py in mdp_code directory to start phase 1 training:

  python main.py   --training_phase  phase_1   --op_phase train   --path_to_data $PATH_TO_DLDP_DATA$
  

  • <i>--path_to_data</i>: path to the directory that has the DLDP data e.g., ./dldp_data_png/
  • The results of the tested models will be saved in the results directory that will be created inside mdp_code
  • The trained model and checkpoints will be saved in ModelCheckpoints after each epoch
  • A TensorBoard log for each training session will be created at logs to provide the visualization and tooling needed to monitor the training

<i>Recall that you might need to change </i>

Phase 2 Training

• Download the final trained model of the first phase used in the main paper i.e., mdp_phase_1_wacv
• Place the downloaded .hdf5 model inside ModelCheckpoints for training
• You need the first phase trained model to strat phase 2 training
• Run main.py in mdp_code directory to start phase 2 training:

  python main.py   --training_phase  phase_2   --op_phase train   --path_to_data $PATH_TO_DPDD_DATA$   --phase_1_checkpoint_model  mdp_phase_1_wacv
  

  • <i>--path_to_data</i>: path to the directory that has the DPDD data e.g., ./dd_dp_dataset_canon/
  • <i>--phase_1_checkpoint_model</i>: the name of the pretrained model from phase 1 e.g., mdp_phase_1_wacv
  • The results of the tested models will be saved in the results directory that will be created inside mdp_code
  • The trained model and checkpoints will be saved in ModelCheckpoints after each epoch
  • A TensorBoard log for each training session will be created at logs to provide the visualization and tooling needed to monitor the training

<i>Recall that you might need to change </i>

Other Training Options

• <i>--patch_size</i>: training patch size
• <i>--img_mini_b</i>: image mini-batch size
• <i>--epoch</i>: number of training epochs
• <i>--lr</i>: initial learning rate
• <i>--schedule_lr_rate</i>: learning rate scheduler (after how many epochs to decrease)
• <i>--bit_depth</i>: image bit depth datatype, 16 for uint16 or 8 for uint8. Recall that we train with 16-bit images
• <i>--dropout_rate</i>: the dropout rate of the conv unit at the network bottleneck

NIMAT Effect

• To generate eight DP-like sub-aperture views (i.e., NIMAT) for each image in a directory, run the following command:

  python main.py   --op_phase nimat   --test_model  mdp_phase_2_wacv   --path_to_data $PATH_TO_DATA$
  

  • <i>--path_to_data</i>: path to the directory that has the images (no ground truth)
  • <i>--downscale_ratio</i>: downscale input test image in case the GPU memory is not sufficient, or use CPU instead
  • The results of the tested models will be saved in the results directory that will be created inside mdp_code

<i>Recall that you might need to change </i>

Contact

Should you have any question/suggestion, please feel free to reach out:

Abdullah Abuolaim (abuolaim@eecs.yorku.ca).

Related Links

• ECCV'18 paper: Revisiting Autofocus for Smartphone Cameras   [project page]
• WACV'20 paper: Online Lens Motion Smoothing for Video Autofocus   [project page]   [presentation]
• ICCP'20 paper: Modeling Defocus-Disparity in Dual-Pixel Sensors   [github]   [presentation]
• ECCV'20 paper: Defocus Deblurring Using Dual-Pixel Data   [project page]   [github]   [presentation]
• ICCV'21 paper: Learning to Reduce Defocus Blur by Realistically Modeling Dual-Pixel Data   [github]   [presentation]
• CVPRW'21 paper: NTIRE 2021 Challenge for Defocus Deblurring Using Dual-pixel Images: Methods and Results   [pdf]   [presentation]
• WACVW'22 paper: Multi-View Motion Synthesis via Applying Rotated Dual-Pixel Blur Kernels   [pdf]   [presentation]

References

[1] Abdullah Abuolaim and Michael S. Brown. Defocus Deblurring Using Dual-Pixel Data. In ECCV, 2020.