Home

Awesome

Ensemble Image Colorization using Convolutional Neural Networks with Refinement Network

(* This research is an extension of Deep Koalarization by Baldassarre et al. Click the link to read the initial research. I also included the original README of the initial research if you want to explore it.)

Abstract

Colorization of grayscale images brings more life to images and presents a more realistic view of the objects present as in the real world. It can be applied on the restoration of historic monochrome images and films as well as assisting cartoon artists. Previous approaches in image colorization involve scribbling color hints on the input grayscale image or choosing a reference color image to be able to transfer the color traits into the input grayscale image. In this study, the burden of requiring user intervention in the colorization process is removed by using a convolutional neural network(CNN) that is trained on a large dataset of color images. Mode cost is evaluated using L2 regression loss wherein the colorized image is compared with the ground truth image to achieve a relatively close colorization. Previous colorization results show effective colorizations on global image semantics like sky, grass, and trees, but suffers from color bleeding where certain colors exceed their assigned regions. Our proposed approach addresses this problem. We also propose a new approach in colorization using CNN by using ensemble colorization where separate colorizations are performed by two different CNNs then a refinement network combines and refines the results. The effectiveness of the approach is further evaluated by performing a "Colorization Turing Test" wherein participants are tasked to determine which of the given set of images are colorized or real.

Full Network Architecture

Colorization Results

(** This part of the README is from the initial research.)

Deep Koalarization

GitHub license GitHub stars GitHub forks arXiv Twitter

Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2 (2017).

Federico Baldassarre*, Diego Gonzalez Morin* and Lucas Rodés Guirao* (* Authors contributed equally)

This project was developed as part of the DD2424 Deep Learning in Data Science course at KTH Royal Institute of Technology, spring 2017.

The code is built using Keras and Tensorflow.

Consider starring this project if you found it useful :star:!

Citation

If you find Deep Koalarization useful in your research, please consider citing our paper as

@article{deepkoal2017,
  author          = {Federico Baldassarre, Diego Gonzalez-Morin, Lucas Rodes-Guirao},
  title           = {Deep-Koalarization: Image Colorization using CNNs and Inception-ResNet-v2},
  journal         = {ArXiv:1712.03400},
  url             = {https://arxiv.org/abs/1712.03400},
  year            = 2017,
  month           = dec
}

arXiv e-print

Abstract

We review some of the most recent approaches to colorize gray-scale images using deep learning methods. Inspired by these, we propose a model which combines a deep Convolutional Neural Network trained from scratch with high-level features extracted from the Inception-ResNet-v2 pre-trained model. Thanks to its fully convolutional architecture, our encoder-decoder model can process images of any size and aspect ratio. Other than presenting the training results, we assess the "public acceptance" of the generated images by means of a user study. Finally, we present a carousel of applications on different types of images, such as historical photographs.

<!--- ## Intro We got the inspiration from the work of Richard Zhang, Phillip Isola and Alexei A. Efros, who realized a network able to colorize black and white images ([blog post](http://richzhang.github.io/colorization/) and [paper](https://arxiv.org/abs/1603.08511)). They trained a network on ImageNet pictures preprocessed to make them gray-scale, with the colored image as the output target. Then we also saw the experiments of Satoshi Iizuka, Edgar Simo-Serra and Hiroshi Ishikawa, who added image classification features to raw pixels fed to the network, improving the overall results ([YouTube review](https://www.youtube.com/watch?v=MfaTOXxA8dM), [blog post](http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/en/) and [paper](http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/data/colorization_sig2016.pdf)). -->

Overview

Inspired by Iizuka and Simo-Serra et al. (2016), we combine a deep CNN architecture with Inception-ResNet-v2 pre-trained on ImageNet dataset, which assists the overall colorization process by extracting high-level features. In particular, Inception-ResNet-v2

<!--- The hidden layers of these models are learned to create a semantic representation of the image that is then used by the final layer (fully connected + softmax) to label the objects in the image. By “cutting” the model at one of its final layers we will get a high dimensional representation of image features, that will be used by our network to perform the colorization task (TensorFlow [tutorial](https://www.tensorflow.org/tutorials/image_retraining) on transfer learning, another [tutorial](https://kwotsin.github.io/tech/2017/02/11/transfer-learning.html) and arXiv [paper](https://arxiv.org/abs/1403.6382)). -->

The fusion between the fixed-size embedding and the intermediary result of the convolutions is performed by means of replication and stacking as described in Iizuka and Simo-Serra et al. (2016).

Fusion

We have used the MSE loss as the objective function.

The Training data for this experiment could come from any source. We decuded to use ImageNet, which nowadays is considered the de-facto reference for image tasks. This way, it makes easier for others to replicate our experiments.

Installation

Refer to INSTRUCTIONS to setup and use the code in this repo.

Results

ImageNet

ImageNet 1

Historical pictures

Historical 1

Thanks to the people who noticed our work!

We are proud if our work gets noticed and helps/inspires other people on their path to knowledge. Here's a list of references we are aware of, some of the authors contacted us, some others we just happened to find online: