Awesome

CLIP for Image-text Matching

A simple pytorch implementation of baseline based-on CLIP for Image-text Matching.

This project provides a CLIP-based training and evaluation framework for image-text matching on MS-COCO dataset.

Requirements

We recommend the following dependencies.

• Python 3.8
• PyTorch 1.7.1
• NumPy (>1.19.5)
• TensorBoard

pip install requirments.txt

Dataset Preparation

COCO Caption

We follow the same split provided by VSE++.

Dataset images can be found here or here. Dataset splits and annotations can be found here.

The final data directory tree should be:

${DATAPATH}/
├── annotations/
│   ├── captions_train2014.json
│   ├── captions_val2014.json
│   ├── coco_train_ids.npy
│   ├── coco_dev_ids.npy
│   ├── coco_test_ids.npy
│   ├──coco_restval_ids.npy
│   └── ...
│          
└── images/ # all images of MS-COCO

Training

You can finetune the model by running:

ViT-B/32:

python main.py --batch_size 256 --epochs 5 --lr 1e-5 --warmup 500 --vision_model ViT-B/32 --dataset_root ${DATAPATH}

ViT-B/16:

python main.py --batch_size 128 --epochs 5 --lr 1e-5 --warmup 500 --vision_model ViT-B/16

Evaluation

You can eval the model by running:

python main.py --eval --resume ${MODELPATH} --vision_model ${VISONMODEL}

Zero-shot Results on MS-COCO

Fine-tuned Results on MS-COCO 5K

Planning

• Mixed precision training
• Providing the training and evaluation codes on Flickr30K.