Awesome
Official repository for the paper
Day-to-Night Image Synthesis for Training Nighttime Neural ISPs @CVPR'22 Oral
Abhijith Punnappurath, Abdullah Abuolaim, Abdelrahman Abdelhamed, Alex Levinshtein, and Michael S. Brown
Samsung Artificial Intelligence Center, Toronto, Canada
[Paper] [arXiv] [Supplemental] [Samsung Reserach blog post]
Citation
If you use this code or the associated data, please cite the paper:
@InProceedings{Punnappurath_2022_CVPR,
author = {Punnappurath, Abhijith and Abuolaim, Abdullah and Abdelhamed, Abdelrahman and Levinshtein, Alex and Brown, Michael S.},
title = {Day-to-night Image Synthesis for Training Nighttime Neural ISPs},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2022},
}
Note
- Results may differ slightly from the numbers reported in the paper because a few images from our original dataset had to be changed due to possible copyright issues. The dataset released here has the same number of images as the original dataset used in the paper; only a small number of images have been replaced.
Get started
- The code was tested on Ubuntu 16.04, Python3.7, CUDA 10.1, cuDNN 8.0, Pytorch 1.7.0
- The code may work in other environments
- Requirements
- opencv-python
- scipy
- exifread
- rawpy
- colour_demosaicing
- matplotlib
- tensorboard
- scikit-image==0.15.0
- Download the dataset [link], and copy to
./dataset/
- (Optional) Download pre-trained models [link], and copy to
./models/
Convert day images to synthetic nighttime images
python3 prepare_data.py --savefoldername night --dim --relight --relight_local
- Synthetic nighttime images will be saved to
synthetic_datasets/night
- Note: While synthesizing our nighttime images, we have also added a few saturated lights to mimic light bulbs in nighttime scenes (see the parameter
num_sat_lights
inprepare_data.py
). Although not mentioned in our paper, we found that adding saturated lights yeilds slightly improved performance - If you wish to skip generating the images, you can download our pre-generated data [link], and copy to
./synthetic_datasets/night
Train Night Mode ISP on synthetic nighttime images
- Clean inputs (Table 1 of the paper)
python3 train.py --data-dir synthetic_datasets/night --which-input clean_raw --wb-illum avg --savefoldername night_clean_raw --on-cuda
- Noisy inputs (Table 2 of the paper)
python3 train.py --data-dir synthetic_datasets/night --which-input noisy_raw --wb-illum avg --savefoldername night_noisy_raw --on-cuda
Test the synthetic nighttime data model
- Clean inputs (Table 1 of the paper)
python3 test.py --model_dir night_clean_raw --set_names iso_50
- Noisy inputs (Table 2 of the paper)
python3 test.py --model_dir night_noisy_raw --set_names iso_1600,iso_3200
Test using our pretrained synthetic nighttime data model
- Clean inputs (Table 1 of the paper)
python3 test.py --model_dir pretrained_night_clean_raw --set_names iso_50
- Noisy inputs (Table 2 of the paper)
python3 test.py --model_dir pretrained_night_noisy_raw --set_names iso_1600,iso_3200
Model | PSNR (dB) | SSIM (dB) |
---|---|---|
Clean raw ISO 50 | 45.97 | 0.9924 |
Noisy raw ISO 1600 | 37.00 | 0.9288 |
Noisy raw ISO 3200 | 36.14 | 0.9182 |
Train and test Night Mode ISP on real data
- Results in last row of Table 1 and Table 2 of the paper
- Uses 3-fold cross validation
- Three separate models have to be trained
- Clean inputs (Table 1 of the paper)
python3 k_fold_split_data.py --which_fold 0 --with_noise 0
python3 train.py --which-input clean_raw --savefoldername real_clean_raw_fold0 --on-cuda
python3 test.py --set_dir data/test --model_dir real_clean_raw_fold0 --set_names iso_50
- Noisy inputs (Table 2 of the paper)
python3 k_fold_split_data.py --which_fold 0 --with_noise 1
python3 train.py --which-input noisy_raw --savefoldername real_noisy_raw_fold0 --on-cuda
python3 test.py --set_dir data/test --model_dir real_noisy_raw_fold0 --set_names iso_1600,iso_3200
- Repeat for folds 1 and 2
Test using our pretrained real data model
- Clean inputs (Table 1 of the paper)
python3 k_fold_split_data.py --which_fold 0 --with_noise 0
python3 test.py --set_dir data/test --model_dir pretrained_real_clean_raw_fold0 --set_names iso_50
- Noisy inputs (Table 2 of the paper)
python3 k_fold_split_data.py --which_fold 0 --with_noise 1
python3 test.py --set_dir data/test --model_dir pretrained_real_noisy_raw_fold0 --set_names iso_1600,iso_3200
- Repeat for folds 1 and 2
Results averaged over 3 folds
Model | PSNR (dB) | SSIM (dB) |
---|---|---|
Clean raw ISO 50 | 46.66 | 0.9923 |
Noisy raw ISO 1600 | 39.25 | 0.9511 |
Noisy raw ISO 3200 | 38.14 | 0.9406 |
Baseline models day
, day_dimmed
, and global_relight
Generate data
day
python3 prepare_data.py --savefoldername day
day_dimmed
python3 prepare_data.py --savefoldername day_dimmed --dim
global_relight
python3 prepare_data.py --savefoldername global_relight --dim --relight
- Choose
<baseline_model_name>
below asday
,day_dimmed
, orglobal_relight
Train and test
- Clean inputs
python3 train.py --data-dir synthetic_datasets/<baseline_model_name> --which-input clean_raw --savefoldername <baseline_model_name>_clean_raw --on-cuda
python3 test.py --model_dir <baseline_model_name>_clean_raw --set_names iso_50
- Noisy inputs
python3 train.py --data-dir synthetic_datasets/<baseline_model_name> --which-input noisy_raw --savefoldername <baseline_model_name>_noisy_raw --on-cuda
python3 test.py --model_dir <baseline_model_name>_noisy_raw --set_names iso_1600,iso_3200
Mixing synthetic and real data
- Uses 3-fold cross validation
- Three separate models have to be trained
- Clean inputs
python3 k_fold_split_data.py --which_fold 0 --with_noise 0
python3 mix_synth_real_data.py --real_percent 10
python3 train.py --which-input clean_raw --savefoldername mix_clean_raw_fold0_percent10 --on-cuda
python3 test.py --set_dir data/test --model_dir mix_clean_raw_fold0_percent10 --set_names iso_50
- Noisy inputs
python3 k_fold_split_data.py --which_fold 0 --with_noise 1
python3 mix_synth_real_data.py --real_percent 10
python3 train.py --which-input noisy_raw --savefoldername mix_noisy_raw_fold0_percent10 --on-cuda
python3 test.py --set_dir data/test --model_dir mix_noisy_raw_fold0_percent10 --set_names iso_1600,iso_3200
- Repeat for folds 1 and 2
(Optional) Full burst dataset:
- Our full burst dataset, containing 30 frames at ISO 50, and 10 frames at ISOs 1600 and 3200, can be downloaded here [link]
- Downloading this dataset is NOT required to reproduce the experiments performed in the paper
- As mentioned in the paper, the bursts can be useful for denoising and other tasks. Please cite our work if you use our dataset
- The real nighttime dataset used for the above experiments can be generated from the full burst dataset by running
python3 generate_real_gt.py
Contact
Abhijith Punnappurath - (abhijith.p@samsung.com; jithuthatswho@gmail.com)