Home

Awesome

artistic-videos

This is the torch implementation for the paper "Artistic style transfer for videos", based on neural-style code by Justin Johnson https://github.com/jcjohnson/neural-style .

Our algorithm allows to transfer the style from one image (for example, a painting) to a whole video sequence and generates consistent and stable stylized video sequences.

Example video:

Artistic style transfer for videos

Contact

For issues or questions related to this implementation, please use the issue tracker. For everything else, including licensing issues, please email us. Our contact details can be found in our paper.

Setup

Tested with Ubuntu 14.04.

Requirements

A fast GPU with a large amount of video memory is recommended to execute this script. The ability to run in CPU mode is impractical due to the enormous running time.

For a resolution of 450x350, you will need at least a 4GB GPU (around 3,5 GB memory usage). If you use cuDNN, a 2GB GPU is sufficient (around 1,7GB memory usage). Memory usage scales linearly with resolution, so if you experience an out of memory error, downscale the video.

Other ways to reduce memory footprint are to use the ADAM optimizer instead of L-BFGS and/or to use the NIN Imagenet model instead of VGG-19. However, we didn't test our method with either of these and you will likely get inferior results.

Simple style transfer

To perform style transfer with mostly the default parameters, execute stylizeVideo.sh <path_to_video> <path_to_style_image>. This script will perform all the steps necessary to create a stylized version of the video. Note: You have to have ffmpeg (or libav-tools for Ubuntu 14.10 and earlier) installed.

FAQ

See here for a list of frequently asked question.

Advanced Usage

Please read the script stylizeVideo.sh to see which steps you have to perform in advance exactly. Basically you have to save the frames of the video as individual image files and you have to compute the optical flow between all adjacent frames as well as the certainty of the flow field (both can be accomplished with makeOptFlow.sh).

There are two versions of this algorithm, a single-pass and a multi-pass version. The multi-pass version yields better results in case of strong camera motion, but needs more iterations per frame.

Basic usage:

th artistic_video.lua <arguments> [-args <fileName>]
th artistic_video_multiPass.lua <arguments> [-args <fileName>]

Arguments can be given by command line and/or written in a file with one argument per line. Specify the path to this file through the option -args. Arguments given by command line will override arguments written in this file.

Basic arguments:

Arguments for the single-pass algorithm (only present in artistic_video.lua)

Arguments for the multi-pass algorithm (only present in artistic_video_multiPass.lua)

Optimization options:

Output options:

Other arguments:

Acknowledgement

Citation

If you use this code or its parts in your research, please cite the following paper:

@TechReport{RuderDB2016,
  author = {Manuel Ruder and Alexey Dosovitskiy and Thomas Brox},
  title = {Artistic style transfer for videos},
  institution  = "arXiv:1604.08610",
  year         = "2016",
}