Awesome
Exploiting Unlabeled Data with Vision and Language Models for Object Detection
Official implementation of Exploiting unlabeled data with vision and language models for object detection.
News
12/29/2023
: Please check ou improved the pseudo labeling with self-training and a split-and-fusion head (paper and code).
Installation
Our project is developed on Detectron2. Please follow the official installation instructions.
Data Preparation
Download the COCO dataset, and put it in the datasets/
directory.
Download our pre-generated pseudo-labeled data, and put them in the datasets/open_voc
directory.
Dataset are organized in the following way:
datasets/
coco/
annotations/
instances_train2017.json
instances_val2017.json
open_voc/
instances_eval.json
instances_train.json
images/
train2017/
000000000009.jpg
000000000025.jpg
...
val2017/
000000000776.jpg
000000000139.jpg
...
Pseudo label generation
If you want to generate and evaluate pseudo labels on your own, please follow our pseudo label generation instruction
Evaluation with pre-trained models
Mask R-CNN:
<table><tbody> <!-- START TABLE --> <!-- TABLE HEADER --> <th valign="bottom">Training Method</th> <th valign="bottom">Novel AP</th> <th valign="bottom">Base AP</th> <th valign="bottom">Overall AP</th> <th valign="bottom">download</th> <!-- TABLE BODY --> <!-- ROW: with LSJ --> <tr><td align="left"><a href="./configs/coco_openvoc_LSJ.yaml">With LSJ</a></td> <td align="center">34.4</td> <td align="center">60.2</td> <td align="center">53.5</td> <td align="center"><a href="https://drive.google.com/file/d/1bk0yrHdPe-gaO0goDDLmHY0KAOy7fR_y/view?usp=drive_link">model</a></td> </tr> <!-- ROW: with out LSJ --> <tr><td align="left"><a href="./configs/coco_openvoc_mask_rcnn.yaml">W/O LSJ</a></td> <td align="center">32.3</td> <td align="center">54.0</td> <td align="center">48.3</td> <td align="center"><a href="https://drive.google.com/file/d/1yaYqWeXzSMDAdCgbKxyTpmu45uv_Xsu-/view?usp=drive_link">model</a></td> </tr> </tbody></table>python -m train_net.py --config configs/coco_openvoc_LSJ.yaml --num-gpus=1 --eval-only --resume
Training
The best model on COCO in the paper is trained with large scale Jitter (LSJ), but training with LSJ requires too many GPU memories. Thus, beside the LSJ version, we also provide training without LSJ.
Training Mask R-CNN with Large Scale Jitter (LSJ).
python train_net.py --config configs/coco_openvoc_LSJ.yaml --num-gpus=8 --use_lsj
Training Mask R-CNN without Large Scale Jitter (LSJ).
python train_net.py --config configs/coco_openvoc_mask_rcnn.yaml --num-gpus=8
Citing VL-PLM
If you use VL-PLM in your work or wish to refer to the results published in this repo, please cite our paper:
@inproceedings{zhao2022exploiting,
title={Exploiting unlabeled data with vision and language models for object detection},
author={Zhao, Shiyu and Zhang, Zhixing and Schulter, Samuel and Zhao, Long and Vijay Kumar, BG and Stathopoulos, Anastasis and Chandraker, Manmohan and Metaxas, Dimitris N},
booktitle={ECCV},
pages={159--175},
year={2022},
organization={Springer}
}