Home

Awesome

ControlGAN

Pytorch implementation for Controllable Text-to-Image Generation. The goal is to generate images from text, and also allow the user to manipulate synthetic images using natural language descriptions, in one framework.

Overview

<img src="archi.jpg" width="900px" height="404px"/>

Controllable Text-to-Image Generation.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, Philip H. S. Torr.<br> University of Oxford <br> In Neural Information Processing Systems, 2019. <br>

Data

  1. Download the preprocessed metadata for bird and coco, and save both into data/
  2. Download bird dataset and extract the images to data/birds/
  3. Download coco dataset and extract the images to data/coco/

Training

All code was developed and tested on CentOS 7 with Python 3.7 (Anaconda) and PyTorch 1.1.

DAMSM model includes text encoder and image encoder

python pretrain_DAMSM.py --cfg cfg/DAMSM/bird.yml --gpu 0
python pretrain_DAMSM.py --cfg cfg/DAMSM/coco.yml --gpu 1

ControlGAN model

python main.py --cfg cfg/train_bird.yml --gpu 2
python main.py --cfg cfg/train_coco.yml --gpu 3

*.yml files include configuration for training and testing.

Pretrained DAMSM Model

Pretrained ControlGAN Model

Testing

python main.py --cfg cfg/eval_bird.yml --gpu 4
python main.py --cfg cfg/eval_coco.yml --gpu 5

Evaluation

Code Structure

Citation

If you find this useful for your research, please use the following.

@article{li2019control,
  title={Controllable text-to-image generation},
  author={Li, Bowen and Qi, Xiaojuan and Lukasiewicz, Thomas and H.~S.~Torr, Philip},
  journal={arXiv preprint arXiv:1909.07083},
  year={2019}
}

Acknowledgements

This code borrows heavily from AttnGAN repository. Many thanks.