Awesome
USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval
Our source code of USER accepted by TIP will be released as soon as possible. It is built on top of the vse_inf in PyTorch.
Data
We organize all data used in the experiments in the same manner as vse_inf:
data
├── coco
│ ├── precomp # pre-computed BUTD region features for COCO, provided by SCAN
│ │ ├── train_ids.txt
│ │ ├── train_caps.txt
│ │ ├── ......
│ │
│ ├── images # raw coco images
│ │ ├── train2014
│ │ └── val2014
│ │
│ └── id_mapping.json # mapping from coco-id to image's file name
│
│
├── f30k
│ ├── precomp # pre-computed BUTD region features for Flickr30K, provided by SCAN
│ │ ├── train_ids.txt
│ │ ├── train_caps.txt
│ │ ├── ......
│ │
│ ├── flickr30k-images # raw coco images
│ │ ├── xxx.jpg
│ │ └── ...
│ └── id_mapping.json # mapping from f30k index to image's file name
│
│
└── vocab # vocab files provided by SCAN (only used when the text backbone is BiGRU)
Training
Train MSCOCO and Flickr30K from scratch:
Modify the corresponding arguments and run train_region_coco.sh
or train_region_f30k.sh
Evaluation
Modify the corresponding arguments in eval.py
and run python eval.py
.
Please use the following bib entry to cite this paper if you are using any resources from the repo.
@article{zhang2024user,
title={USER: Unified semantic enhancement with momentum contrast for image-text retrieval},
author={Zhang, Yan and Ji, Zhong and Wang, Di and Pang, Yanwei and Li, Xuelong},
journal={IEEE Transactions on Image Processing},
year={2024},
publisher={IEEE}
}