Awesome
Introduction
This is Graph Structured Network for Image-Text Matching, source code of GSMN (project page). The paper is accepted by CVPR2020. It is built on top of the SCAN in PyTorch.
Requirements and Installation
We recommended the following dependencies.
- Python 2.7
- PyTorch 1.1.0
- NumPy (>1.12.1)
- TensorBoard
Pretrained model
If you don't want to train from scratch, you can download the pretrained GSMN model from here(for Flickr30K dense model) and here(for Flickr30K sparse model). The performance of this pretrained single model is as follows, in which some Recall@1 values are even better than results produced by our paper:
GSMN-dense:
rsum: 481.4
Average i2t Recall: 87.0
Image to text: 74.4 91.1 95.4 1.0 3.4
Average t2i Recall: 73.5
Text to image: 54.1 79.9 86.5 1.0 9.4
GSMN-sparse:
rsum: 476.8
Average i2t Recall: 86.5
Image to text: 72.8 91.0 95.8 1.0 4.0
Average t2i Recall: 72.4
Text to image: 52.8 78.8 85.6 1.0 10.1
Download data
Download the dataset files. We use the image feature created by SCAN, downloaded here. The text feature, image bounding box and semantic dependency are precomputed, and can be downloaded from here (for Flickr30K and MSCOCO)
Training
python train.py --data_path "$DATA_PATH" --data_name f30k_precomp --vocab_path "$VOCAB_PATH" --logger_name runs/log --model_name "$MODEL_PATH" --bi_gru
Arguments used to train Flickr30K models and MSCOCO models are similar with those of SCAN:
For Flickr30K:
| Method | Arguments |
|---|---|
| GSMN-dense | --max_violation --lambda_softmax=20 --num_epochs=30 --lr_update=15 --learning_rate=.0002 --embed_size=1024 --batch_size=64 |
| GSMN-sparse | --max_violation --lambda_softmax=20 --num_epochs=30 --lr_update=15 --learning_rate=.0002 --embed_size=1024 --batch_size=64 --is_sparse |
For MSCOCO:
| Method | Arguments |
|---|---|
| GSMN-dense | --max_violation --lambda_softmax=10 --num_epochs=20 --lr_update=5 --learning_rate=.0005 --embed_size=1024 --batch_size=32 |
| GSMN-sparse | --max_violation --lambda_softmax=10 --num_epochs=20 --lr_update=5 --learning_rate=.0005 --embed_size=1024 --batch_size=32 --is_sparse |
Evaluation
Test on Flickr30K
python test.py
To do cross-validation on MSCOCO, pass fold5=True with a model trained using
--data_name coco_precomp.
python testall.py
To ensemble sparse model and dense model, specify the model_path in test_stack.py, and run
python test_stack.py
Reference
If you found this code useful, please cite the following paper:
@inproceedings{liu2020graph,
title={Graph Structured Network for Image-Text Matching},
author={Liu, Chunxiao and Mao, Zhendong and Zhang, Tianzhu and Xie, Hongtao and Wang, Bin and Zhang, Yongdong},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={10921--10930},
year={2020}
}