Awesome

Localizing objects using referring expressions

This repository contains code for detecting objects in images mentioned by referring expressions. The code is an implementation of the technique presented in our paper. We have also included links to pretrained models, our split of the Google RefExp dataset and also the processed version of the UNC RefExp dataset.

@inproceedings{nagaraja16refexp,
  title={Modeling Context Between Objects for Referring Expression Understanding},
  author={Varun K. Nagaraja and Vlad I. Morariu and Larry S. Davis},
  booktitle={ECCV},
  year={2016}
}

We have also implemented the baseline and max-margin techniques proposed by Mao et al. in their CVPR 2016 paper. If you use the Google RefExp dataset, please cite this paper

@inproceedings{google_refexp,
  title={Generation and Comprehension of Unambiguous Object Descriptions},
  author={Mao, Junhua and Huang, Jonathan and Toshev, Alexander and Camburu, Oana and Yuille, Alan and Murphy, Kevin},
  booktitle={CVPR},
  year={2016}
}

If you use the UNC RefExp dataset, please cite the following paper

@inproceedings{unc_refexp,
  title={Modeling Context in Referring Expressions},
  author={Licheng Yu and Patric Poirson and Shan Yang and Alexander C. Berg and Tamara L. Berg},
  booktitle={ECCV},
  year={2016}
}

Setup

Clone this repository

git clone --recursive https://github.com/varun-nagaraja/referring-expressions.git

We will call the directory that you cloned this repository into as $RefExp_ROOT

Build external components

Build caffe and pycaffe. The instructions for installing caffe are here.
```
cd $RefExp_ROOT/caffe
make -j8 && make pycaffe
```

Download VGGnet

mkdir $RefExp_ROOT/caffe/models/vggnet
cd $RefExp_ROOT/caffe/models/vggnet
wget https://gist.githubusercontent.com/ksimonyan/211839e770f7b538e2d8/raw/0067c9b32f60362c74f4c445a080beed06b07eb3/VGG_ILSVRC_16_layers_deploy.prototxt
wget http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_16_layers.caffemodel

Build COCO toolbox
```
cd $RefExp_ROOT/coco/PythonAPI
make
```

Download the datasets

The MSCOCO dataset can be download from here. You will need the training images, validation images and object instance annotations.

Our code uses the following directory structure

$COCO_PATH
├── annotations
│   ├── instances_train2014.json
│   └── instances_val2014.json
├── images
│   ├── train2014
│   └── val2014
├── google_refexp
└── unc_refexp

Download Google RefExp dataset with our split and MCG region candidates
```
cd $COCO_PATH
wget https://obj.umiacs.umd.edu/referring-expressions/google_refexp_umd_split.tar.gz
tar -xzf google_refexp_umd_split.tar.gz
rm google_refexp_umd_split.tar.gz
```
Note: If you want the original split of the Google RefExp dataset, follow the instructions at this link. Then move the dataset files to the appropriate folder as indicated above.

Download UNC RefExp dataset with MCG candidates

cd $COCO_PATH
wget https://obj.umiacs.umd.edu/referring-expressions/unc_refexp.tar.gz 
tar -xzf unc_refexp.tar.gz
rm unc_refexp.tar.gz

Testing

Create cache directories where we will store the model and vocabulary files
```
python lib/experiment_settings.py --coco_path $COCO_PATH
```

Download pre-trained vocabulary and model files

cd $COCO_PATH/cache_dir
cd h5_data
wget https://obj.umiacs.umd.edu/referring-expressions/Google_RefExp_vocabulary.txt
wget https://obj.umiacs.umd.edu/referring-expressions/UNC_RefExp_vocabulary.txt
cd ..
cd models
# baseline models trained on Google RefExp and UNC RefExp
wget https://obj.umiacs.umd.edu/referring-expressions/baseline_models.tar.gz
tar -xzf baseline_models.tar.gz
rm baseline_models.tar.gz
# max-margin models
wget https://obj.umiacs.umd.edu/referring-expressions/max_margin_models.tar.gz
tar -xzf max_margin_models.tar.gz
rm max_margin_models.tar.gz
# context models with negative bag margin
wget https://obj.umiacs.umd.edu/referring-expressions/mil_context_withNegMargin_models.tar.gz
tar -xzf mil_context_withNegMargin_models.tar.gz
rm mil_context_withNegMargin_models.tar.gz
# context models with positive bag margin
wget https://obj.umiacs.umd.edu/referring-expressions/mil_context_withPosNegMargin_models.tar.gz
tar -xzf mil_context_withPosNegMargin_models.tar.gz
rm mil_context_withPosNegMargin_models.tar.gz

Note: In the paper, for Google RefExp experiments, we report numbers from models trained on a subset of the training set since we use the remaining training set for validation. However, these pretrained models were trained on the entire training set and hence provide slighty better numbers than those reported in the paper.

Evaluate on a dataset split

python lib/comprehension_experiments.py --coco_path $COCO_PATH --dataset Google_RefExp --exp_name baseline --split_name val --proposal_source gt

Use --visualize option to pause at every image and display localization results

Training

Create files for training
```
python lib/process_dataset.py --coco_path $COCO_PATH --dataset Google_RefExp --exp_name baseline
```
This will first extract region features for all images in the dataset and dump them in a format suitable for loading in caffe. It will require a lot of space on disk depending on the experiment type you want to run.

If you are working with Google RefExp, we will split the training data to create a validation partition of our own. The test set of the Google RefExp dataset is not yet released.

cd $COCO_PATH/cache_dir/h5_data/buffer_16/Google_RefExp_baseline_20
head -n 5038 hdf5_chunk_list.txt > hdf5_chunk_list_part1.txt
tail -n 300 hdf5_chunk_list.txt > hdf5_chunk_list_part2.txt

Edit training prototxt file (Ex.: proto_files/google_refexp/google_refexp.baseline.prototxt) and set the correct source in hdf5_data_param. For example,

hdf5_data_param {
    source: "$COCO_PATH/cache_dir/h5_data/buffer_16/Google_RefExp_train_baseline_20/hdf5_chunk_list_part1.txt"
    batch_size: 16
}

Edit solver.prototxt file (Ex.: proto_files/google_refexp/google_refexp.baseline.solver.prototxt) and set snapshot_prefix appropriately. For example,
```
snapshot_prefix: "$COCO_PATH/cache_dir/models/Google_RefExp_baseline/google_refexp.baseline"
```
Important Create the required model directory (Ex.: $COCO_PATH/cache_dir/models/Google_RefExp_baseline)
Edit training script to use the experiment name you are interested in. Then run the script from the $RefExp_ROOT directory.
```
./proto_files/google_refexp/google_refexp_train.sh
```
The script will print the log to the screen and also write to a file.
When the training is complete, choose the iteration number of the model snapshot with the lowest cross entropy loss on the validation set. Set this iteration number in lib/experiment_settings.py file to test the trained model. The following command will extract the lines which contain the cross entropy loss.
```
grep "Testing net (#1)" -A 4 google_refexp.baseline.log
```