Awesome

Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond

Project Page | Video

This repository provides the official implementation for the following two papers:

Flare7K++

Flare7K++, the first comprehensive nighttime flare removal dataset, consists of 962 real-captured flare images (Flare-R) and 7,000 synthetic flares (Flare7K). Flare7K is generated based on the observation and statistic of real-world nighttime lens flares. It offers 5,000 scattering flare images and 2,000 reflective flare images, consisting of 25 types of scattering flares and 10 types of reflective flares. Flare-R is captured by smartphone rear cameras with different lens contaminants in the dark room. These flare patterns can be randomly added to the flare-free images, forming the flare-corrupted and flare-free image pairs.

Update

• 2023.06.12: We released the training codes and checkpoints for our Flare7K++ [Baidu Netdisk/Google Drive]. More training details can be found at our new technical report: arxiv.
• 2023.06.08: We create a mixing dataset called Flare7K++ that augments the synthetic Flare7K dataset with a new real-captured Flare-R dataset.
• 2023.02.09: Our training code is released.
• 2022.12.28: The MIPI Workshop 2023 is released now. Our dataset serves as a track in this challenge. Please check the CodaLab page to find more details about our challenge.
• 2022.10.12: Upload a flare-corrupted test dataset without ground truth.
• 2022.10.11: Upload the dataset and pretrained model in Baidu Netdisk.
• 2022.10.09: Update baseline inference code for flare removal.
• 2022.09.16: Our paper Flare7K: A Phenomenological Nighttime Flare Removal Dataset is accepted by the NeurIPS 2022 Track Datasets and Benchmarks. 🤗
• 2022.08.27: Update dataloader for our dataset.
• 2022.08.25: Increase the number of test images from 20 to 100. Please download the latest version of our Flare7K dataset.
• 2022.08.19: This repo is created.

Installation

1. Clone the repo

   git clone https://github.com/ykdai/Flare7K.git
   

2. Install dependent packages

   cd Flare7K
   pip install -r requirements.txt
   

3. Install Flare7K<br> Please run the following commands in the Flare7K root path to install Flare7K:<br>

   python setup.py develop
   

Data Download

Paired Data Generation

We provide a on-the-fly dataloader function and a flare-corrupted/flare-free pairs generation script in this repository. To use this function, please put the Flare7K dataset and 24K Flickr dataset on the same path with the generate_flare.ipynb file.

If you only want to generate the flare-corrupted image without reflective flare, you can comment out the following line:

# flare_image_loader.load_reflective_flare('Flare7K','Flare7k/Reflective_Flare')

Pretrained Model

The inference code based on Uformer is released Now. Your can download the pretrained checkpoints from the following links. Please place it under the experiments folder and unzip it, then you can run the test.py for inference. We provide two checkpoints for models training on Flare7K, the model in the folder uformer can help remove both the reflective flares and scattering flares. The uformer_noreflection one can only help remove the scattering flares but is more robust. Now, we prefer the users to test our new model trained on Flare7K++, it can achieve better results and more realistic light source.

Inference Code

To estimate the flare-free images with our checkpoint pretrained on Flare7K++, you can run the test.py or test_large.py (for image larger than 512*512) by using:

python test_large.py --input dataset/Flare7Kpp/test_data/real/input --output result/test_real/flare7kpp/ --model_path experiments/flare7kpp/net_g_last.pth --flare7kpp

If you use our checkpoint pretrained on Flare7K, please run:

python test_large.py --input dataset/Flare7Kpp/test_data/real/input --output result/test_real/flare7k/ --model_path experiments/flare7k/net_g_last.pth

Evaluation Code

To calculate different metrics with our pretrained model, you can run the evaluate.py by using:

python evaluate.py --input result/blend/ --gt dataset/Flare7Kpp/test_data/real/gt/ --mask dataset/Flare7Kpp/test_data/real/mask/

Training model

Training with single GPU

To train a model with your own data/model, you can edit the options/uformer_flare7k_option.yml and run the following codes. You can also add --debug command to start the debug mode:

python basicsr/train.py -opt options/uformer_flare7k_option.yml

If you want to use Flare7K++ for training, please use:

python basicsr/train.py -opt options/uformer_flare7kpp_baseline_option.yml

Training with multiple GPU

You can run the following command for the multiple GPU tranining:

CUDA_VISIBLE_DEVICES=0,1 bash scripts/dist_train.sh 2 options/uformer_flare7k_option.yml

If you want to use Flare7K++ for training, please use:

CUDA_VISIBLE_DEVICES=0,1 bash scripts/dist_train.sh 2 options/uformer_flare7kpp_baseline_option.yml

Flare7k++ structure

├── Flare7K
    ├── Reflective_Flare 
    ├── Scattering_Flare
         ├── Compound_Flare
         ├── Glare_with_shimmer
         ├── Core
         ├── Light_Source
         ├── Streak
├── Flare-R
	├── Compound_Flare
	├── Light_Source
├── test_data
     ├── real
          ├── input
          ├── gt
          ├── mask
     ├── synthetic
          ├── input
          ├── gt
	  ├── mask

In our Flare7K++ dataset, we also update light source annotations for Flare7K dataset, new version contains a tiny glare around the light source to increase the reality. If you want to use the old version, please use the core annotations.

License

This project is licensed under <a rel="license" href="https://github.com/ykdai/Flare7K/blob/main/LICENSE">S-Lab License 1.0</a>. Redistribution and use of the dataset and code for non-commercial purposes should follow this license.

Citation

If you find this work useful, please cite:

@inproceedings{dai2022flare7k,
  title={Flare7K: A Phenomenological Nighttime Flare Removal Dataset},
  author={Dai, Yuekun and Li, Chongyi and Zhou, Shangchen and Feng, Ruicheng and Loy, Chen Change},
  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2022}
}

@inproceedings{dai2023nighttime,
  title={Nighttime Smartphone Reflective Flare Removal using Optical Center Symmetry Prior},
  author={Dai, Yuekun and Luo, Yihang and Zhou, Shangchen and Li, Chongyi and Loy, Chen Change},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2023}
}

@article{dai2023flare7kpp,
      title={Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond}, 
      author={Yuekun Dai and Chongyi Li and Shangchen Zhou and Ruicheng Feng and Yihang Luo and Chen Change Loy},
      year={2023}
}

Contact

If you have any question, please feel free to reach me out at ydai005@e.ntu.edu.sg.