Home

Awesome

Show, Control and Tell

This repository contains the reference code for the paper Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions (CVPR 2019).

Please cite with the following BibTeX:

@inproceedings{cornia2019show,
  title={{Show, Control and Tell: A Framework for Generating Controllable and Grounded Captions}},
  author={Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year={2019}
}

sample results

Environment setup

Clone the repository and create the sct conda environment using the conda.yml file:

conda env create -f conda.yml
conda activate sct

Our code is based on SpeakSee: a Python package that provides utilities for working with Visual-Semantic data, developed by us. The conda enviroment we provide already includes a beta version of this package.

Data preparation

COCO Entities

Download the annotations and metadata file dataset_coco.tgz (~85.6 MB) and extract it in the code folder using tar -xzvf dataset_coco.tgz.

Download the pre-computed features file coco_detections.hdf5 (~53.5 GB) and place it under the datasets/coco folder, which gets created after decompressing the annotation file.

Flickr30k Entities

As before, download the annotations and metadata file dataset_flickr.tgz (~32.8 MB) and extract it in the code folder using tar -xzvf dataset_flickr.tgz.

Download the pre-computed features file flickr30k_detections.hdf5 (~13.1 GB) and place it under the datasets/flickr folder, which gets created after decompressing the annotation file.

Evaluation

To reproduce the results in the paper, download the pretrained model file saved_models.tgz (~4 GB) and extract it in the code folder with tar -xzvf saved_models.tgz.

Sequence controllability

Run python test_region_sequence.py using the following arguments:

ArgumentPossible values
--datasetcoco, flickr
--exp_nameours, ours_without_visual_sentinel, ours_with_single_sentinel
--sample_rlIf used, tests the model with CIDEr optimization
--sample_rl_nwIf used, tests the model with CIDEr + NW optimization
--batch_sizeBatch size (default: 16)
--nb_workersNumber of workers (default: 0)

For example, to reproduce the results of our full model trained on COCO-Entities with CIDEr+NW optimization (Table 2, bottom right), use:

python test_region_sequence.py --dataset coco --exp_name ours --sample_rl_nw  

Set controllability

Run python test_region_set.py using the following arguments:

ArgumentPossible values
--datasetcoco, flickr
--exp_nameours, ours_without_visual_sentinel, ours_with_single_sentinel
--sample_rlIf used, tests the model with CIDEr optimization
--sample_rl_nwIf used, tests the model with CIDEr + NW optimization
--batch_sizeBatch size (default: 16)
--nb_workersNumber of workers (default: 0)

For example, to reproduce the results of our full model trained on COCO-Entities with CIDEr+NW optimization (Table 4, bottom row), use:

python test_region_set.py --dataset coco --exp_name ours --sample_rl_nw  

Expected output

Under logs/, you may also find the expected output of all experiments.

Training procedure

Run python train.py using the following arguments:

ArgumentPossible values
--exp_nameExperiment name
--batch_sizeBatch size (default: 100)
--lrInitial learning rate (default: 5e-4)
--nb_workersNumber of workers (default: 0)
--sample_rlIf used, the model will be trained with CIDEr optimization
--sample_rl_nwIf used, the model will be trained with CIDEr + NW optimization

For example, to train the model with cross entropy, use:

python train.py --exp_name show_control_and_tell --batch_size 100 --lr 5e-4 

To train the model with CIDEr optimization (after training the model with cross entropy), use:

python train.py --exp_name show_control_and_tell --batch_size 100 --lr 5e-5 --sample_rl

To train the model with CIDEr + NW optimization (after training the model with cross entropy), use:

python train.py --exp_name show_control_and_tell --batch_size 100 --lr 5e-5 --sample_rl_nw

Note: the current training code only supports the use of the COCO Entities dataset.

model

COCO Entities

If you want to use only the annotations of our COCO Entities dataset, you can download the annotation file coco_entities_release.json (~403 MB).

The annotation file contains a python dictionary structured as follows:

coco_entities_release.json
 └── <id_image>
      └── <caption>
           └── 'det_sequences'
           └── 'noun_chunks'
           └── 'detections'
           └── 'split'

In details, for each image-caption pair, we provide the following information:

Note that this annotation file includes all image-caption pairs for which at least one noun chunk-detection association has been found. However, in validation and testing phase of our controllable captioning model, we dropped all captions with empty region sets (i.e. those captions with at least one _ in the det_sequences field).

coco entities

By downloading the dataset, you declare that you will use it for research and educational purposes only, any commercial use is prohibited.

Demo

An example of how to use the COCO Entities annotations can be found in the coco_entities_demo.ipynb file.

References

[1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.

[2] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015.

Contact

If you have any general doubt about our work, please use the public issues section on this github repo. Alternatively, drop us an e-mail at marcella.cornia [at] unimore.it or lorenzo.baraldi [at] unimore.it.