Awesome

CCMB and R2D2: A Large-scale Chinese Cross-modal Benchmark and A Vision-Language Framework

🔥🔥🔥 CCMB: A Large-scale Chinese Cross-modal Benchmark (ACM MM 2023)

This repo is the official implementation of CCMB and R2D2</a>.

CCMB is available. It include pre-train dataset (Zero) and 5 downstream datasets. The detailed introduction and download URL are in <font size=3>http://zero.so.com</font>. The 250M data is in <font size=3>https://pan.baidu.com/s/1gnNbjOdCQdqZ4bRNN1S-Vw?pwd=iau8</font>.

R2D2 is a vision-language framework. We release the following code and models:

✅Pre-trained checkpoints.

✅Inference demo.

✅Fine-tuning code and checkpoints for Image-Text Retrieval and Image-Text Matching tasks.

Performance

We show the performance of R2D2_{<font size=1.5>ViT-L</font>} fine-tuned on Flickr30k-CNA dataset. The output of R2D2 is a similarity score between 0 and 1.

Requirements

<pre/>pip install -r requirements.txt</pre>

Pre-trained checkpoints

Fine-tuned checkpoints

Inference demo

• To evaluate the pretrained R2D2 model on image-text pairs, run: <pre>python r2d2_inference_demo.py</pre>
• To evaluate the pretrained PRD2 model on image-text pairs, run: <pre>python prd2_inference_demo.py</pre>

Downstream Tasks

1. Download datasets and pretrained models. for ICR, IQR, ICM, IQM tasks, after downloading you should see the following folder structure:

   ├── IQR_IQM_ICR_ICM_images
   │   
   ├── IQR
   │   ├── train
   │   └── val
   ├── ICR
   │   ├── train
   │   └── val
   ├── IQM
   │   ├── train
   │   └── val
   │── ICM
   │   ├── train
   │   └── val
   for Flickr30k-CNA, after downloading you should see the following folder structure:
   

   ├── Flickr30k-images │
   ├── train │
   ├── val │
   └── test

2. In config/retrieval_*.yaml, set the paths for the dataset and pretrain model paths.
3. Run fine-tuning for the Image-Text Retrieval task.

   sh train_r2d2_retrieval.sh
   

4. Run fine-tuning for the Image-Text Matching task.

   sh train_r2d2_matching.sh
   

Citation

If you find this dataset and code useful for your research, please consider citing.