Home

Awesome

MDETR: Modulated Detection for End-to-End Multi-Modal Understanding

WebsiteColabPaper

This repository contains code and links to pre-trained models for MDETR (Modulated DETR) for pre-training on data having aligned text and images with box annotations, as well as fine-tuning on tasks requiring fine grained understanding of image and text.

We show big gains on the phrase grounding task (Flickr30k), Referring Expression Comprehension (RefCOCO, RefCOCO+ and RefCOCOg) as well as Referring Expression Segmentation (PhraseCut, CLEVR Ref+). We also achieve competitive performance on visual question answering (GQA, CLEVR).

MDETR

TL;DR. We depart from the fixed frozen object detector approach of several popular vision + language pre-trained models and achieve true end-to-end multi-modal understanding by training our detector in the loop. In addition, we only detect objects that are relevant to the given text query, where the class labels for the objects are just the relevant words in the text query. This allows us to expand our vocabulary to anything found in free form text, making it possible to detect and reason over novel combination of object classes and attributes.

For details, please see the paper: MDETR - Modulated Detection for End-to-End Multi-Modal Understanding by Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve and Nicolas Carion.

Aishwarya Kamath and Nicolas Carion made equal contributions to this codebase.

Usage

The requirements file has all the dependencies that are needed by MDETR.

We provide instructions how to install dependencies via conda. First, clone the repository locally:

git clone https://github.com/ashkamath/mdetr.git

Make a new conda env and activate it:

conda create -n mdetr_env python=3.8
conda activate mdetr_env

Install the the packages in the requirements.txt:

pip install -r requirements.txt

Multinode training

Distributed training is available via Slurm and submitit:

pip install submitit

Pre-training

The links to data, steps for data preparation and script for running finetuning can be found in Pretraining Instructions We also provide the pre-trained model weights for MDETR trained on our combined aligned dataset of 1.3 million images paired with text.

The models are summarized in the following table. Note that the performance reported is "raw", without any fine-tuning. For each dataset, we report the class-agnostic box AP@50, which measures how well the model finds the boxes mentioned in the text. All performances are reported on the respective validation sets of each dataset.

<table> <thead> <tr> <th rowspan="2"></th> <th rowspan="2">Backbone</th> <th>GQA</th> <th colspan="2">Flickr</th> <th colspan="4">Refcoco</th> <th rowspan="2"> Url<br></th> <th rowspan="2">Size<br></th> </tr> <tr> <td>AP</td> <td>AP</td> <td>R@1</td> <td>AP</td> <td>Refcoco R@1</td> <td>Refcoco+ R@1</td> <td>Refcocog R@1</td> </tr> </thead> <tbody> <tr> <td>1</td> <td>R101</td> <td>58.9</td> <td>75.6</td> <td>82.5</td> <td>60.3</td> <td>72.1</td> <td>58.0</td> <td>55.7</td> <td><a href="https://zenodo.org/record/4721981/files/pretrained_resnet101_checkpoint.pth?download=1"> model</a></td> <td>3GB</td> </tr> <tr> <td>2</td> <td>ENB3</td> <td>59.5</td> <td>76.6</td> <td>82.9</td> <td>57.6</td> <td>70.2</td> <td>56.7</td> <td>53.8</td> <td><a href="https://zenodo.org/record/4721981/files/pretrained_EB3_checkpoint.pth?download=1">model</a></td> <td>2.4GB</td> </tr> <tr> <td>3</td> <td>ENB5</td> <td>59.9</td> <td>76.4</td> <td>83.7</td> <td>61.8</td> <td>73.4</td> <td>58.8</td> <td>57.1</td> <td><a href="https://zenodo.org/record/4721981/files/pretrained_EB5_checkpoint.pth?download=1">model</a></td> <td>2.7GB</td> </tr> </tbody> </table>

Downstream tasks

Phrase grounding on Flickr30k

Instructions for data preparation and script to run evaluation can be found at Flickr30k Instructions

AnyBox protocol

BackbonePre-training Image DataVal R@1Val R@5Val R@10Test R@1Test R@5Test R@10urlsize
Resnet-101COCO+VG+Flickr82.592.994.983.493.595.3model3GB
EfficientNet-B3COCO+VG+Flickr82.993.295.284.093.895.6model2.4GB
EfficientNet-B5COCO+VG+Flickr83.693.495.184.393.995.8model2.7GB

MergedBox protocol

BackbonePre-training Image DataVal R@1Val R@5Val R@10Test R@1Test R@5Test R@10urlsize
Resnet-101COCO+VG+Flickr82.391.893.783.892.794.4model3GB

Referring expression comprehension on RefCOCO, RefCOCO+, RefCOCOg

Instructions for data preparation and script to run finetuning and evaluation can be found at Referring Expression Instructions

RefCOCO

BackbonePre-training Image DataValTestATestBurlsize
Resnet-101COCO+VG+Flickr86.7589.5881.41model3GB
EfficientNet-B3COCO+VG+Flickr87.5190.4082.67model2.4GB

RefCOCO+

BackbonePre-training Image DataValTestATestBurlsize
Resnet-101COCO+VG+Flickr79.5284.0970.62model3GB
EfficientNet-B3COCO+VG+Flickr81.1385.5272.96model2.4GB

RefCOCOg

BackbonePre-training Image DataValTesturlsize
Resnet-101COCO+VG+Flickr81.6480.89model3GB
EfficientNet-B3COCO+VG+Flickr83.3583.31model2.4GB

Referring expression segmentation on PhraseCut

Instructions for data preparation and script to run finetuning and evaluation can be found at PhraseCut Instructions

BackboneM-IoUPrecision @0.5Precision @0.7Precision @0.9urlsize
Resnet-10153.156.138.911.9model1.5GB
EfficientNet-B353.757.539.911.9model1.2GB

Visual question answering on GQA

Instructions for data preparation and scripts to run finetuning and evaluation can be found at GQA Instructions

BackboneTest-devTest-stdurlsize
Resnet-10162.4861.99model3GB
EfficientNet-B562.9562.45model2.7GB

Long-tailed few-shot object detection

Instructions for data preparation and scripts to run finetuning and evaluation can be found at LVIS Instructions

DataAPAP 50AP rAPcAP furlsize
1%16.725.811.214.619.5model3GB
10%24.238.020.924.924.3model3GB
100%22.535.27.422.725.0model3GB

Synthetic datasets

Instructions to reproduce our results on CLEVR-based datasets are available at CLEVR instructions

<table> <thead> <tr> <th>Overall Accuracy</th> <th>Count</th> <th>Exist<br></th> <th>Compare Number</th> <th>Query Attribute</th> <th>Compare Attribute</th> <th>Url</th> <th>Size</th> </tr> </thead> <tbody> <tr> <td>99.7</td> <td>99.3</td> <td>99.9</td> <td>99.4</td> <td>99.9</td> <td>99.9</td> <td><a href="https://zenodo.org/record/4721981/files/clevr_checkpoint.pth?download=1"> model</a></td> <td>446MB</td> </tr> </tbody> </table>

License

MDETR is released under the Apache 2.0 license. Please see the LICENSE file for more information.

Citation

If you find this repository useful please give it a star and cite as follows! :) :

    @article{kamath2021mdetr,
      title={MDETR--Modulated Detection for End-to-End Multi-Modal Understanding},
      author={Kamath, Aishwarya and Singh, Mannat and LeCun, Yann and Misra, Ishan and Synnaeve, Gabriel and Carion, Nicolas},
      journal={arXiv preprint arXiv:2104.12763},
      year={2021}
    }