Awesome

Annealing-based-Label-Transfer-Learning-for-Open-World-Object-Detection

Introduction

This repository is the official PyTorch implemetation of paper "Annealing-based-Label-Transfer-Learning-for-Open-World-Object-Detection".

NOTE:

In the master branch, we applied our method to the Faster-RCNN framework, and in the ow-detr branch, we applied our method to the same Deformable DETR framework as ow-detr.
If you want to learn more about the disentanglement and the visualization of our approach, please check out the supplementary video.

Key Code

Our key codes of the RCNN-based and DETR-based model are listed below, respectively：

<html> <table style="width: 100%;"> <tr> <td style="width: 50%;">  <pre><code> <strong>RCNN-Based</strong> . └── detectron2 ├── data │ └── LabelTrans_common.py └── modeling ├── meta_arch │ └── rcnn.py └── roi_heads └── AnneallingLT_out.py </code></pre> </td> <td style="width: 50%;">  <pre><code> <strong>DETR-Based</strong> . ├── configs │ └── new1026 ├── main_open_world.py └── models └── AnneallingLT_detr.py ├── requirements.txt └── scripts </code></pre> </td> </tr> </table> </html>

The code of RCNN-based model is built on detectron2 framework. The main structure of the model is set up in the detectron2/modeling/meta_arch/rcnn.py.

In the forming stage, we set the cfg.OWOD.COOLING = False to place the disentanglement degree $\lambda = 0$ and form entangled known proposals. In the extending stage, we simply set the cfg.OWOD.COOLING = True to begin the collaborative learning of known and unknown classes.

Install for RCNN-Based

Requirements

python 3.7, cuda 11.1, torch1.10.1
pip install -r requirements.txt
pip install -e .

Data Preparation for ORE split

You can download the data sets from here and follow these steps to configure the path.
Create folder datasets/VOC2007
Put Annotations and JPEGImages inside datasets/VOC2007
Create folder datasets/VOC2007/ImageSets/Main
Put the content of datasets/OWOD_imagesets inside datasets/VOC2007/ImageSets/Main

Install for DETR-Based

Requirements

We have trained and tested our models on Ubuntu 16.0, CUDA 11.1, GCC 5.4, Python 3.7

conda create -n owdetr python=3.7
conda activate owdetr
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch
pip install -r requirements.txt

Compiling CUDA operators

cd ./models/ops
sh ./make.sh
# unit test (should see all checking is True)
python test.py

Data Preparation for ORE split

You can download the data set from here and follow these steps to configure the path. The files should be organized in the following structure:

OW-DETR/
└── data/
    └── VOC2007/
        └── OWOD/
        	├── JPEGImages
        	├── ImageSets
        	└── Annotations

Pretrained weights

You can download the pre-trained backbone network models and the best OWOD models trained by ours methods for Task t1-t4 here.

Usage

Training

Download the pre-trained backbone network model. R-50.pkl is for faster rcnn framwork and dino_resnet50_pretrain.pth is for ow-detr framwork.
Set the path of the pretrained model in the configs.
You can run train_*.sh in the scripts folder by stages, Where _t*_ represents t1-t4 tasks. Scripts without endings (e.g. train_t2.sh) represent the increment process of forming stage. Scripts with ft endings (e.g. train_t2_ft.sh) represent the fine-tuning process of forming stage, and scripts with _extending endings (e.g. train_t2_extending.sh) represent the extending stage. (Task t1 does not need to be fine-tuned because it has no previously known classes.)
You should run in order such as:

bash scripts/train_t2.sh
bash scripts/train_t2_ft.sh
bash scripts/train_t2_extending.sh

Evaluation

You can sample run test_*.sh in scripts folder.

Citation

If this work helps your research, please consider citing:

@inproceedings{ma2021annealing,
    title={Annealing-based Label-Transfer Learning for Open World Object Detection}, 
    author={Ma, Yuqing and Li, Hainan and Zhang, Zhange and Guo, Jinyang and 
    Zhang, Shanghang and Gong, Ruihao and Liu, Xianglong},
    booktitle={CVPR},
    year={2023}
}