Home

Awesome

Dream-Creator

DOI

This project aims to simplify the process of creating a custom DeepDream model by using pretrained GoogleNet models and custom image datasets.

Here are some example visualizations created with custom DeepDream models trained on summer themed images:

<div align="center"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/big/fc_beachchair.jpg" height="400px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/big/fc_icecream.jpg" height="400px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/big/fc_waterslides.jpg" height="400px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/big/fc_windsurfing.jpg" height="400px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/big/fc_shortshorts.jpg" height="400px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/big/fc_volleyball.jpg" height="400px"> </div> <div align="center"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_flowers.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_hotairballoon.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_jetski.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_sunglasses.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_surfboard.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_tent.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_bike.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_bikini.jpg" height="250px"> <img src="https://raw.githubusercontent.com/ProGamerGov/dream-creator/master/examples/small/fc_volleyball.jpg" height="250px"> </div>

Setup:

Dependencies:

You can find detailed installation instructions for Ubuntu and Windows in the installation guide.

After making sure that PyTorch is installed, you can optionally download the Places365 GoogleNet and Inception5h (InceptionV1) pretrained models with the following command:

python models/download_models.py

If you just want to create DeepDreams with the pretrained models or you downloaded a pretrained model made by someone else with Dream-Creator, then you can skip ahead to visualizing models.

Getting Started

  1. Create & Prepare Your Dataset

    1. Collect Images

    2. Sort images into the required format.

    3. Remove any corrupt images.

    4. Ensure that any duplicates are removed if you have not done so already

    5. Resize the dataset to speed up training.

    6. Calculate the mean and standard deviation of your dataset.

  2. Train a GoogleNet model

  3. Visualize the results

  4. If the results are not great, then you may have to go back to step 1-2 and make some changes with what images, categories, and training parameters are used.

It can take as little as 5 epochs to create visualizations that resemble your training data using the main FC/Logits layer. In order to speed up training and create better looking results, the pretrained BVLC model used is partially frozen in order to protect the lower layers from changing.


Dataset Creation

In order to train a custom DeepDream model, you will need to create a dataset composed of images that you wish to use for training. There are a variety of ways that you can aquire images for your dataset, and you will need at least a couple hundred images for each category/class.

DeepDream is most often performed with image classification models trained on image datasets that are composed of different categories/classes. Image classification models attempt to learn the difference between different image classes and in doing so the neurons gain the ability to create dream-like hallucinations. The images you choose, the differences between them, the differences between your chosen classes, and the number of images used will greatly affect the visualizations that can be produced.

PyTorch image datasets are to be structured where the main directory/folder contains subfolders/directories for each category/class. Below an example of the required dataset structure is shown:

dataset_dir
│
└───category1
│   │   image1.jpg
│   │   image2.jpg
│   │   image3.jpg
│
└───category2
    │   image1.jpg
    │   image2.jpg
    │   image3.jpg

Once you have created your dataset in the proper format, make sure that you remove any duplicate images if you have not done so already. There are a variety of tools that you can use for this task, including free and open source software.

If you have not done so already, you may wish to create a backup copy of your dataset.

Next you will need to verify that none of the images are corrupt in such a way that prevents PyTorch from loading them. To automatically remove any corrupt images from your dataset, use the following command:

python data_tools/remove_bad.py -delete_bad -data_path <training_data>

Next you will likely want to resize your dataset to be closer to the training image size in order to speed up training. Resizing your dataset will not prevent you from creating larger DeepDream images with the resulting model. The included resizing script will only modify images that go above the specified image size with their height or width.

To resize the images in your dataset, use the following command:

python data_tools/resize_data.py -data_path <training_data> -max_size 500

Now with your newly resized dataset, you can calculate the mean and standard deviation of your dataset for use in training, and DeepDreaming. Make sure to recalculate the mean and standard deviation again if you modify the dataset by adding or removing images.

To calculate the mean and standard deviation of your dataset, use the following command and save the output for the next step:

python data_tools/calc_ms.py -data_path <training_data>

Now you can start training your DeepDream model by running the GoogleNet training script. It's recommended that you save the model every 5-10 epochs in order to monitor the quality of the visualizations.

After training your models, you can add a color correlation matrix to them for color decorrelation with the following command:

python data_tools/calc_cm.py -data_path <training_data> -model_file <bvlc_out120>.pth

GoogleNet Training

Basic training command:

python train_googlenet.py -data_path <training_data> -balance_classes -batch_size 96 -data_mean <mean> -data_sd <sd>

Input options:

Training options:

Model options:

Output options:

Other options:

Dataset options:


Visualizing Results

Visualizing GoogleNet FC Layer Results

After training a new DeepDream model, you'll need to test it's visualizations. The best visualizations are found in the main FC layer also known as the 'logits' layer. This script helps you quickly and easily visualize all of a specified layer's channels in a particular model for a particular model epoch, by generating a separate image for each channel.

Input options:

Processing options:

Only Required If Model Doesn't Contain Them, Options:

Output options:

Other options:

Basic FC (logits) layer visualization:

python vis_multi.py -model_file <bvlc_out120>.pth

Advanced FC (logits) layer visualization:

python vis_multi.py -model_file <bvlc_out120>.pth -layer fc -color_decorrelation -fft_decorrelation -random_scale -random_rotation -lr 0.4 -output_dir <output_dir> -padding 16 -jitter 16,8

Performing DeepDream With Your Newly Trained Model

This script lets you create DeepDream hallucinations with trained GoogleNet models.

Input options:

Only Required If Model Doesn't Contain Them, Options:

Output options:

Tiling options:

Other options:

Basic DeepDream:

python vis.py -model_file <bvlc_out120>.pth -layer mixed5a

Advanced DeepDream:

python vis.py -model_file <bvlc_out120>.pth -layer mixed5a/conv_5x5_relu -channel 9 -color_decorrelation -fft_decorrelation -random_scale -random_rotation -lr 0.4 -padding 16 -jitter 16,8

Dataset Cleaning + Building & Visualization Tools

See here for more information on all the included scripts/tools relating to dataset creation, cleaning, and preparation.