Awesome

Learning Linear Transformations for Fast Image and Video Style Transfer

Prerequisites

Pytorch
torchvision
opencv for video generation

All code tested on Ubuntu 16.04, pytorch 0.4.1, and opencv 3.4.2

Style Transfer

Clone from github: git clone https://github.com/sunshineatnoon/LinearStyleTransfer
Download pre-trained models from google drive.
Uncompress to root folder :

cd LinearStyleTransfer
unzip models.zip
rm models.zip

Artistic style transfer

python TestArtistic.py

or conduct style transfer on relu_31 features

python TestArtistic.py --vgg_dir models/vgg_r31.pth --decoder_dir models/dec_r31.pth --matrixPath models/r31.pth --layer r31

Photo-realistic style transfer

For photo-realistic style transfer, we need first compile the pytorch_spn repository.

cd libs/pytorch_spn
sh make.sh
cd ../..

Then:

python TestPhotoReal.py

Note: images with _filtered.png as postfix are images filtered by the SPN after style transfer, images with _smooth.png as postfix are images post process by a smooth filter.

Video style transfer

python TestVideo.py

Real-time video demo

python real-time-demo.py --vgg_dir models/vgg_r31.pth --decoder_dir models/dec_r31.pth --matrixPath models/r31.pth --layer r31

Model Training

Data Preparation

MSCOCO

wget http://msvocds.blob.core.windows.net/coco2014/train2014.zip

WikiArt
- Either manually download from kaggle.
- Or install kaggle-cli and download by running:
```
kg download -u <username> -p <password> -c painter-by-numbers -f train.zip
```

Training

Train a style transfer model

To train a model that transfers relu4_1 features, run:

python Train.py --vgg_dir models/vgg_r41.pth --decoder_dir models/dec_r41.pth --layer r41 --contentPath PATH_TO_MSCOCO --stylePath PATH_TO_WikiArt --outf OUTPUT_DIR

or train a model that transfers relu3_1 features:

python Train.py --vgg_dir models/vgg_r31.pth --decoder_dir models/dec_r31.pth --layer r31 --contentPath PATH_TO_MSCOCO --stylePath PATH_TO_WikiArt --outf OUTPUT_DIR

Key hyper-parameters:

style_layers: which features to compute style loss.
style_weight: larger style weight leads to heavier style in transferred images.

Intermediate results and weight will be stored in OUTPUT_DIR

Train a SPN model to cancel distortions for photo-realistic style transfer

Run:

python TrainSPN.py --contentPath PATH_TO_MSCOCO

Acknowledgement

We use the smooth filter by LouieYang in the photo-realistic style transfer.

Citation

@inproceedings{li2018learning,
    author = {Li, Xueting and Liu, Sifei and Kautz, Jan and Yang, Ming-Hsuan},
    title = {Learning Linear Transformations for Fast Arbitrary Style Transfer},
    booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
    year = {2019}
}