Home

Awesome

<div align="center">

Composed Image Retrieval on Real-life Images

This repository contains the Composed Image Retrieval on Real-life images (CIRR) dataset.

For details please see our ICCV 2021 paper - Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models.

<sup> You are currently viewing the Dataset repository. Site navigation > Project homepage  |  Code repository </sup>

arXiv arXiv

</div>

News and Upcoming Updates

Download CIRR Dataset

Our dataset is structured in a similar way as Fashion-IQ, an existing dataset on this task. The files include annotations, raw images, and the optional pre-extracted image features.

Annotations

Obtain the annotations by:

# create a `data` folder at your desired location
mkdir data
cd data

# clone the cirr_dataset branch to the local data/cirr folder
git clone -b cirr_dataset git@github.com:Cuberick-Orion/CIRR.git cirr

The data/cirr folder contains all relevant annotations. The file structure is described below.

Raw Images

Updated October 2024 -- Please contact us if you are having trouble gaining access to the raw images from NLVR2.

Starting from late 2023, we have been made aware by multiple research groups that the NLVR2 team is not responding to their requests. To this end, please see the following steps in obtaining the raw images:

  1. Please first contact the NLVR team and fill out a Google form agreeing to their terms of service. Instructions are here.
  2. If you receive no response from the NLVR team, email us.
  3. When contacting us, please explicitly state that you have filled out the NLVR team's Google form agreeing to their terms of service.

[!IMPORTANT] The NLVR2 repository provides another way to obtain the images, which is to download the images by URLs. But we do not recommend it, as many of the links are broken, and the downloaded files lack the sub-folder structure in the /train folder.

Instead, please follow the above instruction to directly download the raw images.

Pre-extracted Image Features

The available types of image features are:

Each zip file we provide contains a folder of individual image feature files .pkl.

Once downloaded, unzip it into data/cirr/, following the file structure below.

Dataset File Structure

<details> <summary>The downloaded dataset should look like this (click to expand)</summary>
data
└─── cirr
    ├─── captions
    │        cap.VER.test1.json
    │        cap.VER.train.json
    │        cap.VER.val.json
    ├─── captions_ext
    │        cap.ext.VER.test1.json
    │        cap.ext.VER.train.json
    │        cap.ext.VER.val.json
    ├─── image_splits
    │        split.VER.test1.json
    │        split.VER.train.json
    │        split.VER.val.json
    ├─── img_raw  
    │    ├── train
    │    │    ├── 0 # sub-level folder structure inherited from NLVR2 (carries no special meaning in CIRR)
    │    │    │    <IMG0_ID>.png
    │    │    │    <IMG0_ID>.png
    │    │    │         ...
    │    │    ├── 1
    │    │    │    <IMG0_ID>.png
    │    │    │    <IMG0_ID>.png
    │    │    │         ...
    │    │    ├── 2
    │    │    │    <IMG0_ID>.png
    │    │    │    <IMG0_ID>.png
    │    │    └──       ...
    │    ├── dev         
    │    │      <IMG0_ID>.png
    │    │      <IMG1_ID>.png
    │    │           ...
    │    └── test1       
    │           <IMG0_ID>.png
    │           <IMG1_ID>.png
    │                ...
    ├─── img_feat_res152 
    │        <Same subfolder structure as above>
    └─── img_feat_frcnn         
             <Same subfolder structure as above>
</details>

Dataset File Description

Test-split Evaluation Server

We do not publish the ground truth for the test split of CIRR. Instead, an evaluation server is hosted here, should you prefer to publish results on the test-split. The functions of the test-split server will be incrementally updated.

See test-split server instructions.

The server is hosted independently at CECS ANU, so please email us if the site is down.

License

Citation

Please cite our paper if it helps your research:

@InProceedings{Liu_2021_ICCV,
    author    = {Liu, Zheyuan and Rodriguez-Opazo, Cristian and Teney, Damien and Gould, Stephen},
    title     = {Image Retrieval on Real-Life Images With Pre-Trained Vision-and-Language Models},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {2125-2134}
}

Contact

If you have any questions regarding our dataset, model, or publication, please create an issue in the project repository, or email us.