Home

Awesome

SketchySceneColorization - SIGA2019

Paper | Supplementary Material | Project Page

This repository hosts the datasets and the code for the SketchyScene Colorization system (SIGGRAPH Asia 2019). Please refer to our paper for more information: Language-based Colorization of Scene Sketches.

examples

System Overview

Our system supports two-mode interactive colorization for a given input scene sketch and text-based colorization instructions, using three models, namely, the instance matching model, foreground colorization model, and background colorization model.

system

Outline

Requirements

Preparations

Please follow the instructions in the following three sections (Instance Matching, Foreground Instance Colorization, and Background Colorization) to download the dataset and pre-trained models and place them in the correct directories.

Instance Matching

For the details of MATCHING dataset and the code, please refer to the Instance_Matching directory.

matching

Foreground Instance Colorization

For the details of FOREGROUND dataset and the code, please refer to the Foreground_Instance_Colorization directory.

foreground

Background Colorization

For the details of BACKGROUND dataset and the code, please refer to the Background_Colorization directory.

background

Colorizing With The Whole Pipeline

Our system allows users to colorize the sketches through language instructions. If the result is not satisfactory, users can also withdraw the last instruction.

:fire: We have provided some test examples in examples directory.

  1. To colorize a sketch, run the command like:

    python3 sketchyscene_colorization_main.py --image_id 9996 \
                                          --instruction 'the bus is orange with gray windows'
    • Set image_id to the sketch you want.
    • Try other instructions by changing the instruction.

    You will see the results in outputs directory.

  2. To withdraw the last instruction, run the command like:

    python3 sketchyscene_colorization_main.py --command 'withdraw' --image_id 9996

    See what happens in outputs directory :)

examples

Citation

Please cite the corresponding paper if you found the datasets or code useful:

@article{zouSA2019sketchcolorization,
  title = {Language-based Colorization of Scene Sketches},
  author = {Zou, Changqing and Mo, Haoran and Gao, Chengying and Du, Ruofei and Fu, Hongbo},
  journal = {ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH Asia 2019)},
  year = {2019},
  volume = 38,
  number = 6,
  pages = {233:1--233:16}
}

@inproceedings{zou2018sketchyscene,
  title={Sketchyscene: Richly-annotated scene sketches},
  author={Zou, Changqing and Yu, Qian and Du, Ruofei and Mo, Haoran and Song, Yi-Zhe and Xiang, Tao and Gao, Chengying and Chen, Baoquan and Zhang, Hao},
  booktitle={Proceedings of the european conference on computer vision (ECCV)},
  pages={421--436},
  year={2018}
}