Home

Awesome

ELD

The implementation of CVPR 2020 (Oral) paper "A Physics-based Noise Formation Model for Extreme Low-light Raw Denoising" and its journal (TPAMI) version "Physics-based Noise Modeling for Extreme Low-light Photography". Interested readers are also referred to an insightful Note about this work in Zhihu (Chinese).

:sparkles: News

Highlights

<img src="imgs/pipeline.png" height="140px"/>

<img src="imgs/capture_setup.jpg" height="140px"/> <img src="imgs/example_images.png" height="140px"/>

<img src="imgs/results.png" height="140px"/>

Prerequisites

Notice this codebase relies on my own customized rawpy, which provides more functionalities than the official one. This is released together with our datasets and the pretrained models (See GoogleDrive or Baidudisk (0lby)) To build rawpy from source, please first compile and install the LibRaw library following the official instructions, then type pip install -e . in the customized rawpy directory.

Quick Start

Due to the business license, we are unable to to provide the noise model as well as the calibration method. Instead, we release our collected ELD dataset and our pretrained models to facilitate future research.

To reproduce our results presented in the paper (Table 1 and 2), please take a look at scripts/test_SID.sh and scripts/test_ELD.sh

Update: (2022-01-08) We release the training code and the synthetic datasets per the users' requests. The training scripts and the user instructions can be found in scripts/train.sh. Additionally, we provide the baseline noise models (G/G+P/G+P*) and the calibrated noise parameters for all cameras of ELD for training (see noise.py and train_syn.py), which could serve as a starting point to develop your own noise model.

We use lmdb to prepare datasets, please refer to util/lmdb_data.py to see how we generate datasets from SID. We also provide a new implementation of a classic radiometric calibration method EMoR, and utilize it to calibrate the CRF of SonyA7S2, which could be further used to simulate realistic on-board ISP as in the commercial SonyA7S2 camera.

ELD Dataset

The dataset capture protocol is shown as follow:

<img src="imgs/dataset.png" height="250px"/>

We choose three ISO settings (800, 1600, 3200) and four low light factors (x1, x10, x100, x200) to capture the dataset (x1/x10 is not used in our paper). Image ids 1, 6, 11, 16 represent the long-exposure reference images. Please refer to ELDEvalDataset class in data/sid_dataset.py for more details.

Citation

If you find our code helpful in your research or work please cite our paper.

@article{wei2021physics,
  title={Physics-based noise modeling for extreme low-light photography},
  author={Wei, Kaixuan and Fu, Ying and Zheng, Yinqiang and Yang, Jiaolong},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
  volume={44},
  number={11},
  pages={8520--8537},
  year={2021},
  publisher={IEEE}
}

@inproceedings{wei2020physics,
  title={A Physics-based Noise Formation Model for Extreme Low-light Raw Denoising},
  author={Wei, Kaixuan and Fu, Ying and Yang, Jiaolong and Huang, Hua},
  booktitle={IEEE Conference on Computer Vision and Pattern Recognition},
  year={2020},
}

Contact

If you find any problem, please feel free to contact me (kxwei at princeton.edu kaixuan_wei at bit.edu.cn). A brief self-introduction (including your name, affiliation and position) is required, if you would like to get an in-depth help from me. I'd be glad to talk with you if more information (e.g. your personal website link) is attached. Note I would not reply to any impolite/aggressive email that violates the above criteria.