Home

Awesome

show-adapt-and-tell

This is the official code for the paper

Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner <br> Tseng-Hung Chen, Yuan-Hong Liao, Ching-Yao Chuang, Wan-Ting Hsu, Jianlong Fu, Min Sun <br> To appear in ICCV 2017

<div align='center'> <img src='images/teaser.png' height="405px"> </div>

In this repository we provide:

If you find this code useful for your research, please cite

@article{chen2017show,
  title={Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner},
  author={Chen, Tseng-Hung and Liao, Yuan-Hong and Chuang, Ching-Yao and Hsu, Wan-Ting and Fu, Jianlong and Sun, Min},
  journal={arXiv preprint arXiv:1705.00930},
  year={2017}
}

Requirements

P.S. Please clone the repository with the --recursive flag:

# Make sure to clone with --recursive
git clone --recursive https://github.com/tsenghungchen/show-adapt-and-tell.git

Data Preprocessing

MSCOCO Captioning dataset

Feature Extraction

  1. Download the pretrained ResNet-101 model and place it under data-prepro/MSCOCO_preprocess/resnet_model/.
  2. Please modify the caffe path in data-prepro/MSCOCO_preprocess/extract_resnet_coco.py.
  3. Go to data-prepro/MSCOCO_preprocess and run the following script: ./download_mscoco.sh for downloading images and extracting features.

Captions Tokenization

  1. Clone the NeuralTalk2 repository and head over to the coco/ folder and run the IPython notebook to generate a json file for Karpathy split: coco_raw.json.
  2. Run the following script: ./prepro_mscoco_caption.sh for downloading and tokenizing captions.
  3. Run python prepro_coco_annotation.py to generate annotation json file for testing.

CUB-200-2011 with Descriptions

Feature Extraction

  1. Run the script ./download_cub.sh to download the images in CUB-200-2011.
  2. Please modify the input/output path in data-prepro/MSCOCO_preprocess/extract_resnet_coco.py to extract and pack features in CUB-200-2011.

Captions Tokenization

  1. Download the description data.
  2. Run python get_split.py to generate dataset split following the ECCV16 paper "Generating Visual Explanations".
  3. Run python prepro_cub_annotation.py to generate annotation json file for testing.
  4. Run python CUB_preprocess_token.py for tokenization.

Models from the paper

Pretrained Models

Download all pretrained and adaption models:

Example Results

Here are some example results where the captions are generated from these models:

<table border=1> <tr> <td> <img src='images/im6795.jpg' height="185px"> </br> MSCOCO: A large air plane on a run way. </br> CUB-200-2011: A large white and black airplane with a large beak. </br> TGIF: A plane is flying over a field. </br> Flickr30k: A large airplane is sitting on a runway. </td> <td> <img src='images/im11063.jpg' height="185px"> </br> MSCOCO: A traffic light is seen in front of a large building. </br> CUB-200-2011: A yellow traffic light with a yellow light. </br> TGIF: A traffic light is hanging on a pole. </br> Flickr30k: A street sign is lit up in the dark </td> </tr> </table> <table border=1> <tr> <td> <img src='images/im22197.jpg' height="185px"> </br> MSCOCO: A black dog sitting on the ground next to a window. </br> CUB-200-2011: A black and white dog with a black head. </br> TGIF: A dog is looking at something in the mirror. </br> Flickr30k: A black dog is looking out of the window. </td> <td> <img src='images/im270.jpg' height="185px"> </br> MSCOCO: A man riding a skateboard up the side of a ramp. </br> CUB-200-2011: A man riding a skateboard on a white ramp. </br> TGIF: A man is doing a trick on a skateboard. </br> Flickr30k: A man in a blue shirt is doing a trick on a skateboard. </td> </tr> </table>

Training

The training codes are under the show-adapt-tell/ folder.

Simply run python main.py for two steps of training:

Training the source model with paired image-caption data

Please set the Boolean value of "G_is_pretrain" to True in main.py to start pretraining the generator.

Training the cross-domain captioner with unpaired data

After pretraining, set "G_is_pretrain" to False to start training the cross-domain model.

License

Free for personal or research use; for commercial use please contact me.