Home

Awesome

<div align="center"> <h2>BoxTeacher</h2> <h4>Exploring High-Quality Pseudo Masks for Weakly Supervised Instance Segmentation</h4> <br> <a href="https://scholar.google.com/citations?hl=zh-CN&user=PH8rJHYAAAAJ">Tianheng Cheng</a><sup><span>1</span></sup>, <a href="https://xinggangw.info/">Xinggang Wang</a><sup><span>1</span></sup>, <a href="https://scholar.google.com/citations?user=PIeNN2gAAAAJ&hl=zh-CN&oi=ao">Shaoyu Chen</a><sup><span>1</span></sup>, <a href="https://scholar.google.com/citations?user=pCY-bikAAAAJ&hl=zh-CN">Qian Zhang</a><sup><span>2</span></sup>, <a href="http://eic.hust.edu.cn/professor/liuwenyu/"> Wenyu Liu</a><sup><span>1,&#8224;</span></sup> </br>

<sup>1</sup> School of EIC, HUST, <sup>2</sup> Horizon Robotics <br> (<span></span>: corresponding author)

<div> <a href="https://arxiv.org/abs/2210.05174">[arXiv Preprint (Camera Ready)]</a> </div> </div>

Updates

Highlights

Overview

Labeling objects with pixel-wise segmentation requires a huge amount of human labor compared to bounding boxes. Most existing methods for weakly supervised instance segmentation focus on designing heuristic losses with priors from bounding boxes. While, we find that box-supervised methods can produce some fine segmentation masks and we wonder whether the detectors could learn from these fine masks while ignoring low-quality masks. To answer this question, we present BoxTeacher, an efficient and end-to-end training framework for high-performance weakly supervised instance segmentation, which leverages a sophisticated teacher to generate high-quality masks as pseudo labels. Considering the massive noisy masks hurt the training, we present a mask-aware confidence score to estimate the quality of pseudo masks and propose the noise-aware pixel loss and noise-reduced affinity loss to adaptively optimize the student with pseudo masks.

<div align="center"> <img width="80%" src="assets/main.png"> <br> <br> </div>

Models

COCO Instance Segmentation

ModelBackboneScheduleAPAP<sup>test</sup>WeightsLog
BoxTeacherR-501x32.632.9ckptslog*
BoxTeacherR-503x34.935.0ckptslog*
BoxTeacherR-1013x36.236.5ckptslog*
BoxTeacherR-101-DCN3x37.237.6ckpts-
BoxTeacherSwin-B3x40.240.5ckpts-

Quick start

Preliminaries

BoxTeacher is mainly developed based on detectron2 and Adelaidet.

  1. Install dependencies for BoxTeacher.
# install detectron2
python setup.py build develop

# install adelaidet
cd AdelaiDet
python setup.py build develop
cd ..
  1. Prepare the datasets for BoxTeacher.
boxteacher
datasets/
 - coco/
 - voc/
 - cityscapes/

You can refer to detectron-doc for more details about (custom) datasets.

  1. Prepare the pre-trained weights for different backbones.
mkdir pretrained_models
cd pretrained_models
# download the weights with the links from the above table.

Training

python train_net.py --config-file <path/to/config> --num-gpus 8

Testing

python train_net.py --config-file <path/to/config> --num-gpus 8 --eval MODEL.WEIGHTS <path/to/weights>

A simple guide for applying BoxTeacher to other methods

  1. create the wrapper class (BoxTeacher).

  2. modify the instance segmentation method (e.g., CondInst, Mask2Former) by:

  1. train and evaluate.

Acknowledgements

BoxTeacher is based on detectron2 and Adelaidet and we sincerely thanks for their code and contribution to the community!

License

BoxTeacher is released under the MIT Licence.

Citation

If you find BoxTeacher is useful in your research or applications, please consider giving us a star 🌟 and citing BoxTeacher by the following BibTeX entry.

@inproceedings{Cheng2022BoxTeacher,
  title     =   {BoxTeacher: Exploring High-Quality Pseudo Labels for Weakly Supervised Instance Segmentation},
  author    =   {Cheng, Tianheng and Wang, Xinggang and Chen, Shaoyu and Zhang, Qian and Liu, Wenyu},
  booktitle =   {{IEEE/CVF} Conference on Computer Vision and Pattern Recognition,
               {CVPR} 2022, New Orleans, LA, USA, June 18-24, 2022},
  year      =   {2022}
}