Awesome

This paper was accepted at CVPR 2022!

Slow description video.

This repo provides code for QB-Norm (Cross Modal Retrieval with Querybank Normalisation)

Usage example

python dynamic_inverted_softmax.py --sims_train_test_path msrvtt/tt-ce-train-captions-test-videos-seed0.pkl --sims_test_path msrvtt/tt-ce-test-captions-test-videos-seed0.pkl --test_query_masks_path msrvtt/tt-ce-test-query_masks.pkl

To test QB-Norm on your own data you need to:

1. Extract the similarity matrix between the caption from the training split and the videos from the testing split path/to/sims/train/test
2. Extract testing split similarity matrix (similarities between testing captions and testing video) path/to/sims/test
3. Run QB-Norm

python dynamic_inverted_softmax.py --sims_train_test_path path/to/sims/train/test --sims_test_path path/to/sims/test

Data

The similarity matrices for each method were extracted using the official repositories as follows: CE+, TT-CE+, CLIP2Video, CLIP4Clip (for CLIP4Clip we used the official repo to train from scratch new models since they do not provide pre-trained weights), CLIP, MMT, Audio-Retrieval.

Here you can find our trained weights for CLIP4Clip: MSRVTT, DiDeMo, LSMDC, Activity-Net.

You can download the extracted similarity matrices for training and testing here: MSRVTT, MSRVTT 1kA CLIP2Video, MSVD, DiDeMo, LSMDC.

Text-Video retrieval results

The value used for the inverse temperature is 20, with the exception for CLIP2Video where we used 1/1.99.

QB-Norm Results on MSRVTT Benchmark

QB-Norm Results on MSVD Benchmark

QB-Norm Results on DiDeMo Benchmark

QB-Norm Results on LSMDC Benchmark

The temperature used for CLIP4Clip method on the LSMDC dataset is 0.8.

QB-Norm Results on VaTeX Benchmark

QB-Norm Results on QuerYD Benchmark

Text-Image retrieval results

QB-Norm Results on MSCoCo Benchmark

Text-Audio retrieval results

QB-Norm Results on AudioCaps Benchmark

References

If you find this code useful or use the extracted similarity matrices, please consider citing:

@inproceedings{bogolin2021cross,
      title={Cross Modal Retrieval with Querybank Normalisation}, 
      author={Simion-Vlad Bogolin and Ioana Croitoru and Hailin Jin and Yang Liu and Samuel Albanie},
      booktitle={CVPR}
      year={2022}
}