Home

Awesome

layout2im

This repository provides a PyTorch implementation of layout2im, which can generate images from layouts.

<p align="center"><img src='images/layout2im.jpg' width='650px'></p>

Online demo can be found here.

<p align='center'><img src='images/demo.gif' width='1000px'></p>

Paper

Image Generation from Layout <br/> Bo Zhao, Lili Meng,Weidong Yin, Leonid Sigal<br/> The Univerisity of British Columbia, Vector Institute <br/> IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, (<b>Oral</b>)

Network Architechture

Overview of our Layout2Im networks for generating images from layout during training.

<p align='center'><img src='images/pipeline.jpg' width='1000px'></p>

Usage

1. Creating virtual environment (optional)

All code was developed and tested on Ubuntu 18.04 with Python 3.6 (Anaconda) and PyTorch 1.0.

$ conda create -n layout2im python=3.6
$ conda activate layout2im

2. Install COCO API

$ cd ~
$ git clone https://github.com/cocodataset/cocoapi.git
$ cd cocoapi/PythonAPI/
$ python setup.py install
$ cd ..

3. Cloning the repository

$ git clone git@github.com:zhaobozb/layout2im.git
$ cd layout2im

4. Installing dependencies

$ pip install -r requirements.txt

5. Downloading datasets

To download COCO dataset to datasets/coco:

$ bash scripts/download_coco.sh

To download Visual Genome dataset to datasets/vg and preprocess it:

$ bash scripts/download_vg.sh
$ python scripts/preprocess_vg.py

6. Downloading trained models

Download the trained models to checkpoints/pretrained/.

  1. trained model on COCO
  2. trained model on Visual Genome

7. Testing

Testing on COCO dataset:

$ python layout2im/test.py --dataset coco --coco_dir datasets/coco \
                           --saved_model checkpoints/pretrained/netG_coco.pt \
                           --results_dir checkpoints/pretrained_results_coco

Testing on Visual Genome dataset:

$ python layout2im/test.py --dataset vg --coco_dir datasets/vg \
                           --saved_model checkpoints/pretrained/netG_vg.pt \
                           --results_dir checkpoints/pretrained_results_vg

8. Training

$ python layout2im/train.py

The training script has a number of command-line flags that you can use to configure the model architecture, hyperparameters, and input / output settings:

Results

1. Images generated from layouts

<p align='center'><img src='images/results_coco.jpg' width='1000px'></p>

2. Diverse results generated from the same layout

<p align='center'><img src='images/results_diverse.jpg' width='1000px'></p>

3. Examples of interactive image generation

<p align='center'><img src='images/results_inter.jpg' width='1000px'></p>

Citation

If you find this code useful in your research then please cite

@inproceedings{zhaobo2019layout2im,
  title={Image Generation from Layout},
  author={Zhao, Bo and Meng, Lili and Yin, Weidong and Sigal, Leonid},
  booktitle={CVPR},
  year={2019}
}

Acknowledgement

Our project borrows some source files from sg2im. We thank the authors. This research was partially supported by NSERC Discovery, NSERC DAS, NSERC CFI grants and Mitacs GlobaLink Award. We gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan V GPU used for this research.