Home

Awesome

Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond

Project Page | Video

<img src="assets/flare7k.png" width="800px"/>

This repository provides the official implementation for the following two papers:

<p> <div><strong>Flare7K: A Phenomenological Nighttime Flare Removal Dataset</strong></div> <div><a href="https://ykdai.github.io/">Yuekun Dai</a>, <a href="https://li-chongyi.github.io/">Chongyi Li</a>, <a href="https://shangchenzhou.com/">Shangchen Zhou</a>, <a href="https://jnjaby.github.io/">Ruicheng Feng</a>, <a href="https://www.mmlab-ntu.com/person/ccloy/">Chen Change Loy</a></div> <div>Accepted to <strong>NeurIPS 2022</strong></div><div><a href="https://arxiv.org/abs/2210.06570"> arXiv </a> </p> <p> <div><strong>Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond</strong></div> <div><a href="https://ykdai.github.io/">Yuekun Dai</a>, <a href="https://li-chongyi.github.io/">Chongyi Li</a>, <a href="https://shangchenzhou.com/">Shangchen Zhou</a>, <a href="https://jnjaby.github.io/">Ruicheng Feng</a>, <a href="https://github.com/Luo-Yihang/">Yihang Luo</a>, <a href="https://www.mmlab-ntu.com/person/ccloy/">Chen Change Loy</a></div> <div>Accepted to <strong>TPAMI, 2024</strong></div> <div><a href="https://arxiv.org/abs/2306.04236"> arXiv </a> </p>

Flare7K++

Flare7K++, the first comprehensive nighttime flare removal dataset, consists of 962 real-captured flare images (Flare-R) and 7,000 synthetic flares (Flare7K). Flare7K is generated based on the observation and statistic of real-world nighttime lens flares. It offers 5,000 scattering flare images and 2,000 reflective flare images, consisting of 25 types of scattering flares and 10 types of reflective flares. Flare-R is captured by smartphone rear cameras with different lens contaminants in the dark room. These flare patterns can be randomly added to the flare-free images, forming the flare-corrupted and flare-free image pairs.

Update

Installation

  1. Clone the repo

    git clone https://github.com/ykdai/Flare7K.git
    
  2. Install dependent packages

    cd Flare7K
    pip install -r requirements.txt
    
  3. Install Flare7K<br> Please run the following commands in the Flare7K root path to install Flare7K:<br>

    python setup.py develop
    

Data Download

Baidu NetdiskGoogle DriveNumberDescription
Flare7K++(new)linklink7,962Flare7K++ consists of Flare7K and Flare-R. Flare7K offers 5,000 scattering flare images and 2,000 reflective flare images, consisting of 25 types of scattering flares and 10 types of reflective flares. Flare-R offers 962 real-captured flare patterns.
Background Imageslinklink23,949The background images are sampled from [Single Image Reflection Removal with Perceptual Losses, Zhang et al., CVPR 2018]. We filter our most of the flare-corrupted images and overexposed images.
Flare-corrupted imageslinklink645We offer an extra flare-corrupted dataset without ground truth. It contains 645 images captured by different cameras and some images are very challenging.

Paired Data Generation

We provide a on-the-fly dataloader function and a flare-corrupted/flare-free pairs generation script in this repository. To use this function, please put the Flare7K dataset and 24K Flickr dataset on the same path with the generate_flare.ipynb file.

If you only want to generate the flare-corrupted image without reflective flare, you can comment out the following line:

# flare_image_loader.load_reflective_flare('Flare7K','Flare7k/Reflective_Flare')

Pretrained Model

The inference code based on Uformer is released Now. Your can download the pretrained checkpoints from the following links. Please place it under the experiments folder and unzip it, then you can run the test.py for inference. We provide two checkpoints for models training on Flare7K, the model in the folder uformer can help remove both the reflective flares and scattering flares. The uformer_noreflection one can only help remove the scattering flares but is more robust. Now, we prefer the users to test our new model trained on Flare7K++, it can achieve better results and more realistic light source.

Training DataBaidu NetdiskGoogle Drive
Flare7Klinklink
Flare7K++ (new)linklink

Inference Code

To estimate the flare-free images with our checkpoint pretrained on Flare7K++, you can run the test.py or test_large.py (for image larger than 512*512) by using:

python test_large.py --input dataset/Flare7Kpp/test_data/real/input --output result/test_real/flare7kpp/ --model_path experiments/flare7kpp/net_g_last.pth --flare7kpp

If you use our checkpoint pretrained on Flare7K, please run:

python test_large.py --input dataset/Flare7Kpp/test_data/real/input --output result/test_real/flare7k/ --model_path experiments/flare7k/net_g_last.pth

Evaluation Code

To calculate different metrics with our pretrained model, you can run the evaluate.py by using:

python evaluate.py --input result/blend/ --gt dataset/Flare7Kpp/test_data/real/gt/ --mask dataset/Flare7Kpp/test_data/real/mask/

Training model

Training with single GPU

To train a model with your own data/model, you can edit the options/uformer_flare7k_option.yml and run the following codes. You can also add --debug command to start the debug mode:

python basicsr/train.py -opt options/uformer_flare7k_option.yml

If you want to use Flare7K++ for training, please use:

python basicsr/train.py -opt options/uformer_flare7kpp_baseline_option.yml

Training with multiple GPU

You can run the following command for the multiple GPU tranining:

CUDA_VISIBLE_DEVICES=0,1 bash scripts/dist_train.sh 2 options/uformer_flare7k_option.yml

If you want to use Flare7K++ for training, please use:

CUDA_VISIBLE_DEVICES=0,1 bash scripts/dist_train.sh 2 options/uformer_flare7kpp_baseline_option.yml

Flare7k++ structure

├── Flare7K
    ├── Reflective_Flare 
    ├── Scattering_Flare
         ├── Compound_Flare
         ├── Glare_with_shimmer
         ├── Core
         ├── Light_Source
         ├── Streak
├── Flare-R
	├── Compound_Flare
	├── Light_Source
├── test_data
     ├── real
          ├── input
          ├── gt
          ├── mask
     ├── synthetic
          ├── input
          ├── gt
	  ├── mask

In our Flare7K++ dataset, we also update light source annotations for Flare7K dataset, new version contains a tiny glare around the light source to increase the reality. If you want to use the old version, please use the core annotations.

License

This project is licensed under <a rel="license" href="https://github.com/ykdai/Flare7K/blob/main/LICENSE">S-Lab License 1.0</a>. Redistribution and use of the dataset and code for non-commercial purposes should follow this license.

Citation

If you find this work useful, please cite:

@inproceedings{dai2022flare7k,
  title={Flare7K: A Phenomenological Nighttime Flare Removal Dataset},
  author={Dai, Yuekun and Li, Chongyi and Zhou, Shangchen and Feng, Ruicheng and Loy, Chen Change},
  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2022}
}

@inproceedings{dai2023nighttime,
  title={Nighttime Smartphone Reflective Flare Removal using Optical Center Symmetry Prior},
  author={Dai, Yuekun and Luo, Yihang and Zhou, Shangchen and Li, Chongyi and Loy, Chen Change},
  booktitle = {Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  year = {2023}
}

@article{dai2023flare7kpp,
      title={Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyond}, 
      author={Yuekun Dai and Chongyi Li and Shangchen Zhou and Ruicheng Feng and Yihang Luo and Chen Change Loy},
      year={2023}
}

Contact

If you have any question, please feel free to reach me out at ydai005@e.ntu.edu.sg.