Awesome

[CVPR 2024] MeaCap: Memory-Augmented Zero-shot Image Captioning

Authors: Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Zhengjue Wang, Bo Chen <br/> official implementation of MeaCap.

Catalogue:

• <a href='#introduction'>Introduction</a>
• <a href='#citation'>Citation</a>
• <a href='#data'>Data Preparation</a>
  • <a href='#environment'>Environment</a>
  • <a href='#memory'>Memory bank</a>
  • <a href='#model'>Model Zoo</a>
• <a href='#inference'>Inference</a>
  • <a href='#tf'>Training free</a>
  • <a href='#tot'>Text-only training</a>
  • <a href='#other'>Memory concepts + ViECAP</a>
• <a href='#experiments'>Experiments</a>
• <a href='#acknowledgments'>Acknowledgments</a>

Introduction

Zero-shot image captioning (IC) without well-paired image-text data can be divided into two categories, training-free and text-only-training. The main difference between them is whether using a textual corpus to train the LM. Though achieving attractive performance w.r.t. some metrics, existing methods often exhibit some common drawbacks. Training-free methods tend to produce hallucinations, while text-only-training often lose generalization capability. To move forward, in this paper, we propose a novel Memory-Augmented zero-shot image Captioning framework (MeaCap). Specifically, equipped with a textual memory, we introduce a retrieve-then-filter module to get key concepts that are highly related to the image. By deploying our proposed memory-augmented visual-related fusion score in a keywords-to-sentence LM, MeaCap can generate concept-centered captions that keep high consistency with the image with fewer hallucinations and more world-knowledge.

Citation

If you think MeaCap is useful, please cite this paper!

@article{zeng2024meacap,
  title={MeaCap: Memory-Augmented Zero-shot Image Captioning},
  author={Zeng, Zequn and Xie, Yan and Zhang, Hao and Chen, Chiyu and Wang, Zhengjue and Chen, Bo},
  journal={arXiv preprint arXiv:2403.03715},
  year={2024}
}

Data Preparation

Environment

Prepare the python environment:

pip install -r requirements.txt

Memory bank

We have preprocessed textual corpus of CC3M, SS1M, COCO, and Flickr30k and transformed them into CLIP and SentenceBERT embeddings for fast retrieval. Download our preprocessed memory files and put them into ./data/memory/ , as:

data
└── memory
    ├── cc3m
    │   ├── memory_captions.json
    │   ├── memory_clip_embeddings.pt
    │   └── memory_wte_embeddings.pt
    ├── coco
    │   ├── memory_captions.json
    │   ├── memory_clip_embeddings.pt
    │   └── memory_wte_embeddings.pt
    └── ...    

you can also preprocess a new textual memory bank, for example:

python prepare_embedding.py --memory_id coco --memory_path data/memory/coco/memory_captions.json

Model Zoo

MeaCap use multiple pretrained models to finish different purposes. The default version of language model is CBART. We provide the download link of pretrained CBART and caption-finetuned CBART. Please download these weights and put them into ./checkpoints/ .

If you want to finetune CBART on your own caption corpus, please follow the official training instruction from CBART.

Inference

Training-free

For training-free version $MeaCap_{TF}$, we use a pretrained CBART. To bridge the gap between pretrained dataset one-billion-word and caption-style texts, we use a default prompt "The image depicts that". We also support prompt ensembling by setting --prompt_ensembling.

python inference.py --use_prompt  --memory_id cc3m --img_path ./image_example --lm_model_path ./checkpoints/CBART_one_billion 

Text-only-training

For text-only-training version $MeaCap_{ToT}$, we use finetuned CBART where prompts are needless.

python inference.py --memory_id coco --img_path ./image_example --lm_model_path ./checkpoints/CBART_COCO 

Memory concepts + ViECAP

We also supporting add memory concepts to strong baseline ViECAP in a plug-and-play way, namely $MeaCap_{InvLM}$. We simply need to replace the entity module by our proposed retrieve-then-filter module in the inference stage and then the performance can be improved. Details are shown in Appendix of our paper.

python viecap_inference.py --memory_id coco --image_path "*.jpg" --weight_path "checkpoints/train_coco/coco_prefix-0014.pt"

Experiments

Zero-shot captioning

In/Cross-domain captioning

Acknowledgements

This code is heavily depend on ConZIC, CBART and ViECAP.

Thanks for their good work.