Home

Awesome

White-Balance Emulator for Color Augmentation

What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance

Mahmoud Afifi<sup>1</sup> and Michael S. Brown<sup>1,2</sup> <br></br><sup>1</sup>York University    <sup>2</sup>Samsung AI Center (SAIC) - Toronto

Project page - Paper - Supplementary Materials

ICCV_github_teaser

Our augmentation method can accurately emulate realistic color constancy degradation. Existing color augmentation methods often generate unrealistic colors which rarely happen in reality (e.g., green skin or purple grass). More importantly, the visual appearance of existing color augmentation techniques does not well represent the color casts produced by incorrect WB applied onboard cameras, as shown below.

ICCV_github_examples

Quick start

1. Python:

  1. Requirements: numpy & Pillow
  1. Run wbAug.py; examples:
  1. demo.py shows an example of how to use the WBEmulator module

2. PyTorch:

torch_demo.py shows an example of how to use the WBEmulator module to augment images on the fly. This example can be easily adapted for TensorFlow as well. This code applies the WB augmentation to each image loaded from the data loader. See the implementation of BasicDataset in torch_demo.py as an example of PyTorch data loader with our WB augmenter. The parameter aug_prob controls the probability of applying the WB augmenter.

The demo in torch_demo.py uses the ImageNET pre-trained model of VGG19. The demo uses a single image of a hamster (located in \images and shown below) and randomly applies one of our WB/camera profiles to the image before classifying.

example

As shown below, the network has a different prediction each time because of the WB effect. The same BasicDataset class in torch_demo.py could be adjusted to use in training phase.

Output:
toy poodle 0.15292471647262573
hen 0.5484228134155273
toy poodle 0.15292471647262573
hen 0.5260483026504517
hen 0.5484228134155273
hen 0.5260483026504517
hamster 0.2579324543476105
hen 0.5260483026504517
hamster 0.2579324543476105
toy poodle 0.15292471647262573

In this example, we showed how to use the WB augmenter in a custom database class. To see how to use the WB augmenter for PyTorch built-in datasets, please see this Colab example. In this example, we use this repo, which provides a combined version of the WB augmenter Python implementation to facilitate cloning to Colab. We used CIFAR-10 to train a simple network with and without the WB augmenter. As shown in the Colab example, training with the WB augmenter results in +3% improvement compared to training on original data.

3. Matlab:

View WB color augmenter on File Exchange

  1. Run install_.m
  2. Try our demos:
    • demo_single_image to process signle image
    • demo_batch to process an image directory
    • demo_WB_color_augmentation to process an image directory and repeating the corresponding ground truth files for our generated images
    • demo_GUI (located in GUI directory) for a GUI interface
  3. To use the WB augmenter inside your code, please follow the following steps:
    • Either run install_() or addpath to code/model directories:
     addpath('src');
     addpath('models'); 
     %or use install_()
    

Dataset

We used images from Set1 of the Rendered WB dataset to build our method.

Cat-2_testing_set_CIFAR10_classes

In our paper, we introduced a new testing set that contains CIFAR-10 classes to evaluate trained models with different settings. This testing set contains 15,098 rendered images that reflect real in-camera WB settings. Our testing set is divided into ten directories, each includes testing images for one of CIFAR-10 classes. You can download our testing set from the following links: 32x32 pixels | 224x224 pixels | 227x227 pixels

MIT License

Publication

If you use this code or our dataset, please cite our paper:

Mahmoud Afifi and Michael S. Brown. What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance. International Conference on Computer Vision (ICCV), 2019.

@InProceedings{Afifi_2019_ICCV,
author = {Afifi, Mahmoud and Brown, Michael S.},
title = {What Else Can Fool Deep Learning? Addressing Color Constancy Errors on Deep Neural Network Performance},
booktitle = {The IEEE International Conference on Computer Vision (ICCV)},
month = {October},
year = {2019}
}

Related Research Projects