Awesome
Dense Distinct Query for End-to-End Object Detection (CVPR2023)
By Shilong Zhang*, Xinjiang Wang*, Jiaqi Wang, Jiangmiao Pang, Chengqi Lyu, Wenwei Zhang, Ping Luo, Kai Chen
Updates
21/3/2023: DDQ DETR(12 epoch 52.1 mAP) has been released at ddq_detr branch. Please run git checkout ddq_detr to use it.
20/3/2023: DDQ FCN and DDQ R-CNN has been released at main branch
Abstract:
One-to-one label assignment in object detection has successfully obviated the need of non-maximum suppression (NMS) as a postprocessing
and makes the pipeline end-to-end. However, it triggers a new dilemma as the widely used sparse queries cannot guarantee a high recall, while dense queries inevitably bring more similar queries and encounters optimization difficulty. As both sparse and dense queries are problematic, then what are the expected queries in end-to-end object detection? This paper shows that the solution should be Dense Distinct Queries (DDQ). Concretely, we first lay dense queries like traditional detectors and then select distinct ones for one-to-one assignments. DDQ blends the advantages of traditional and recent end-to-end detectors and significantly improves the performance of various detectors including FCN, R-CNN, and DETRs. Most impressively, ddq_detr achieves 52.1 AP on MS-COCO dataset within 12 epochs using a ResNet-50 backbone, outperforming all existing detectors in the same setting. DDQ also shares the benefit of end-to-end detectors in crowded scenes and achieves 93.8 AP on CrowdHuman. We hope DDQ can inspire researchers to consider the complementarity between traditional methods and end-to-end detectors.
NOTE
-
The DDQ FCN/RCNN was implemented based on mmdetection2.0, which has been released in this main branch .
-
DDQ DETR is implemented based on mmdetection3.0. You need to switch to the ddq_detr branch of this repo.
All our implementation can be find in projects directory.
Install
DDQ FCN&R-CNN
DDQ FCN and DDQ R-CNN has been fully tested under MMDetection V2.22.0 and MMCV V1.4.7 under torch 1.9.0+cu111, other versions may not be compatible. We include the necessary source code of MMCV and MMDetection in this repo, you can build the MMCV by following command.
cd mmcv-1.4.7
MMCV_WITH_OPS=1 python setup.py build_ext --inplace
ln -s mmcv-1.4.7/mmcv ./
cd ..
export PYTHONPATH=`pwd`:$PYTHONPATH
DDQ DETR
DDQ DETR has been fully tested under MMDetection V3.0.0rc6, MMCV v2.0.0rc4 and MMEngine v0.6.0 under torch 1.9.0+cu111, other versions may not be compatible. We include the necessary source code of MMCV, MMDetection and MMEngine in the ddq_detr branch of this repo, you just need to build the MMCV by following command.
git check ddq_detr
cd mmcv-2.0.0rc4
MMCV_WITH_OPS=1 python setup.py build_ext --inplace
ln -s mmcv-2.0.0rc4/mmcv ./
cd ..
export PYTHONPATH=`pwd`:$PYTHONPATH
Prepare the dataset
DDQ
├── data
│ ├── coco
│ │ ├── annotations
│ │ │ ├──instances_train2017.json
│ │ │ ├──instances_val2017.json
│ │ ├── train2017
│ │ ├── val2017
Train and Test
8 is the number of gpus.
For slurm
Train
GPUS=8 sh tools/slurm_train.sh partition_name job_name projects/configs/ddq_fcn/ddq_fcn_r50_1x.py ./exp/ddq-fcn
Test
GPUS=8 sh tools/slurm_test.sh partition_name job_name projects/configs/ddq_fcn/ddq_fcn_r50_1x.py path_to_checkpoint --eval bbox
For Pytorch distributed
Train
sh tools/train.sh projects/configs/ddq_fcn/ddq_fcn_r50_1x.py 8 ./exp/ddq_fcn
Test
sh tools/test.sh projects/configs/ddq_fcn/ddq_fcn_r50_1x.py path_to_checkpoint 8 --eval bbox
Results & Checkpoints
We find that the performance is unstable and may fluctuate by about 0.2 mAP.
| Model | Backbone | Lr schd | Augmentation | box AP(val) | Model | log |
|---|---|---|---|---|---|---|
| DDQ FCN | R-50 | 12e | Normal | 41.5 | ckpt | log |
| DDQ FCN | R-50 | 36e | DETR | 44.8 | ckpt | log |
| DDQ R-CNN | R-50 | 12e | Normal | 44.6 | ckpt | log |
| DDQ R-CNN | R-50 | 36e | DETR | 48.1 | ckpt | log |
Please git checkout ddq_detr and follow the install instruction to use following checkpoint
| Model | Backbone | Lr schd | Augmentation | box AP(val) | Model | log |
|---|---|---|---|---|---|---|
| DDQ DETR-4scale | R-50 | 12e | DETR | 51.3 | ckpt | log |
| DDQ DETR-5scale | R-50 | 12e | DETR | 52.1 | ckpt | log |
| DDQ DETR-4scale | Swin-L | 30e | DETR | 58.7 | ckpt | log |
Citations
Please cite our paper in your publications if it helps your research:
@InProceedings{Zhang_2023_CVPR,
author = {Zhang, Shilong and Wang, Xinjiang and Wang, Jiaqi and Pang, Jiangmiao and Lyu, Chengqi and Zhang, Wenwei and Luo, Ping and Chen, Kai},
title = {Dense Distinct Query for End-to-End Object Detection},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2023},
pages = {7329-7338}
}