Home

Awesome

deep-photo-styletransfer

Code and data for paper "Deep Photo Style Transfer"

Disclaimer

This software is published for academic and non-commercial use only.

Setup

This code is based on torch. It has been tested on Ubuntu 14.04 LTS.

Dependencies:

CUDA backend:

Download VGG-19:

sh models/download_models.sh

Compile cuda_utils.cu (Adjust PREFIX and NVCC_PREFIX in makefile for your machine):

make clean && make

Usage

Quick start

To generate all results (in examples/) using the provided scripts, simply run

run('gen_laplacian/gen_laplacian.m')

in Matlab or Octave and then

python gen_all.py

in Python. The final output will be in examples/final_results/.

Basic usage

  1. Given input and style images with semantic segmentation masks, put them in examples/ respectively. They will have the following filename form: examples/input/in<id>.png, examples/style/tar<id>.png and examples/segmentation/in<id>.png, examples/segmentation/tar<id>.png;
  2. Compute the matting Laplacian matrix using gen_laplacian/gen_laplacian.m in Matlab. The output matrix will have the following filename form: gen_laplacian/Input_Laplacian_3x3_1e-7_CSR<id>.mat;

Note: Please make sure that the content image resolution is consistent for Matting Laplacian computation in Matlab and style transfer in Torch, otherwise the result won't be correct.

  1. Run the following script to generate segmented intermediate result:
th neuralstyle_seg.lua -content_image <input> -style_image <style> -content_seg <inputMask> -style_seg <styleMask> -index <id> -serial <intermediate_folder>
  1. Run the following script to generate final result:
th deepmatting_seg.lua -content_image <input> -style_image <style> -content_seg <inputMask> -style_seg <styleMask> -index <id> -init_image <intermediate_folder/out<id>_t_1000.png> -serial <final_folder> -f_radius 15 -f_edge 0.01

You can pass -backend cudnn and -cudnn_autotune to both Lua scripts (step 3. and 4.) to potentially improve speed and memory usage. libcudnn.so must be in your LD_LIBRARY_PATH. This requires cudnn.torch.

Image segmentation

Note: In the main paper we generate all comparison results using automatic scene segmentation algorithm modified from DilatedNet. Manual segmentation enables more diverse tasks hence we provide the masks in examples/segmentation/.

The mask colors we used (you could add more colors in ExtractMask function in two *.lua files):

Color variableRGB ValueHex Value
blue0 0 2550000ff
green0 255 000ff00
black0 0 0000000
white255 255 255ffffff
red255 0 0ff0000
yellow255 255 0ffff00
grey128 128 128808080
lightblue0 255 25500ffff
purple255 0 255ff00ff

Here are some automatic and manual tools for creating a segmentation mask for a photo image:

Automatic:

Manual:

Examples

Here are some results from our algorithm (from left to right are input, style and our output):

<p align='center'> <img src='examples/input/in3.png' height='194' width='290'/> <img src='examples/style/tar3.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_3.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in4.png' height='194' width='290'/> <img src='examples/style/tar4.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_4.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in13.png' height='194' width='290'/> <img src='examples/style/tar13.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_13.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in9.png' height='194' width='290'/> <img src='examples/style/tar9.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_9.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in20.png' height='194' width='290'/> <img src='examples/style/tar20.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_20.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in1.png' height='194' width='290'/> <img src='examples/style/tar1.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_1.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in39.png' height='194' width='290'/> <img src='examples/style/tar39.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_39.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in57.png' height='194' width='290'/> <img src='examples/style/tar57.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_57.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in47.png' height='194' width='290'/> <img src='examples/style/tar47.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_47.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in58.png' height='194' width='290'/> <img src='examples/style/tar58.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_58.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in51.png' height='194' width='290'/> <img src='examples/style/tar51.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_51.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in7.png' height='194' width='290'/> <img src='examples/style/tar7.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_7.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in23.png' width='290'/> <img src='examples/input/in23.png' width='290'/> <img src='examples/final_results/best23_t_1000.png' width='290'/> </p> <p align='center'> <img src='examples/input/in16.png' height='194' width='290'/> <img src='examples/style/tar16.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_16.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in30.png' height='194' width='290'/> <img src='examples/style/tar30.png' height='194' width='290'/> <img src='examples/refine_posterization/refine_30.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in2.png' height='194' width='290'/> <img src='examples/style/tar2.png' height='194' width='290'/> <img src='examples/final_results/best2_t_1000.png' height='194' width='290'/> </p> <p align='center'> <img src='examples/input/in11.png' width='290'/> <img src='examples/style/tar11.png' width='290'/> <img src='examples/refine_posterization/refine_11.png' width='290'/> </p>

Acknowledgement

Citation

If you find this work useful for your research, please cite:

@article{luan2017deep,
  title={Deep Photo Style Transfer},
  author={Luan, Fujun and Paris, Sylvain and Shechtman, Eli and Bala, Kavita},
  journal={arXiv preprint arXiv:1703.07511},
  year={2017}
}

Contact

Feel free to contact me if there is any question (Fujun Luan fl356@cornell.edu).