Home

Awesome

Weakly Supervised Instance Segmentation using the Bounding Box Tightness Prior

[PDF] [BibTeX] [Supplemental]

This project hosts the code for the implementation of Weakly Supervised Instance Segmentation using the Bounding Box Tightness Prior (NeurIPS 2019).

The main code is based on maskrcnn-benchmark (#5c44ca7) and the post-processing code is based on meanfield-matlab.

Introduction

This paper presents a weakly supervised instance segmentation method that consumes training data with tight bounding box annotations. The major difficulty lies in the uncertain figure-ground separation within each bounding box since there is no supervisory signal about it. We address the difficulty by formulating the problem as a multiple instance learning (MIL) task, and generate positive and negative bags based on the sweeping lines of each bounding box. The proposed deep model integrates MIL into a fully supervised instance segmentation network, and can be derived by the objective consisting of two terms, i.e., the unary term and the pairwise term. The former estimates the foreground and background areas of each bounding box while the latter maintains the unity of the estimated object masks. The experimental results show that our method performs favorably against existing weakly supervised methods and even surpasses some fully supervised methods for instance segmentation on the PASCAL VOC dataset.

Installation

Check INSTALL.md for installation instructions.

Dataset

All details of dataset construction can be found in Sec 4.1 of our paper.

Training

We construct the training set by two following settings:

Testing

We evaluate our method on PASCAL VOC 2012 validation set. (COCO format download link)

Note that the conversion of annotated format from VOC to COCO will result in inaccurate segment boundaries. See Evaluation for more details.

Format and Path

In our experiment, we convert the generated dataset into COCO format.

Before the training, please modified paths_catalog.py and enter the correct data path for voc_2012_aug_train_cocostyle, voc_2012_val_cocostyle, and voc_2012_coco_aug_train_cocostyle.

Training

Run the bash files directly:

Training on PASCAL VOC (Augmented) with 4 GPUs

bash train_voc_aug.sh

Training on PASCAL VOC (Augmented) + COCO with 4 GPUs

train_voc_coco_aug.sh

or type the bash commands:

Training on PASCAL VOC (Augmented) with 4 GPUs

python -m torch.distributed.launch --nproc_per_node=4 tools/train_net.py --config-file ./configs/BBTP/e2e_mask_rcnn_R_101_FPN_4x_voc_aug_cocostyle.yaml

Training on PASCAL VOC (Augmented) + COCO with 4 GPUs

python -m torch.distributed.launch --nproc_per_node=4 tools/train_net.py --config-file ./configs/BBTP/e2e_mask_rcnn_R_101_FPN_4x_voc_coco_aug_cocostyle.yaml

All bash commands are derived from maskrcnn-benchmark (#5c44ca7).

You may also want to see the original README.md of maskrcnn-benchmark.

Evaluation

Although COCO dataset has its own python API for evaluation, the conversion of annotated format from VOC to COCO will result in inaccurate segment boundaries.

To avoid such issue, we recommend to evaluate the predicted results via standard VOC API directly by the following steps:

1. Save the predictions

2. Evaluate

(Optional) 3. Post-processing (DenseCRF)

Result

We provide the model weights and mask files of all experiments in this section.

Reported in the main paper

DatasetmAP@0.25mAP@0.50mAP@0.70mAP@0.75Post-processingModel
PASCAL VOC (Augmented)74.753.723.616.9w/o DenseCRFlink
PASCAL VOC (Augmented)75.058.930.421.6w/ DenseCRF-
PASCAL VOC (Augmented) + COCO76.854.423.717.4w/o DenseCRFlink
PASCAL VOC (Augmented) + COCO77.260.129.421.2w/ DenseCRF-

Reproduced with the released code

DatasetmAP@0.25mAP@0.50mAP@0.70mAP@0.75Post-processingModel
PASCAL VOC (Augmented)74.054.124.517.1w/o DenseCRFlink
PASCAL VOC (Augmented)74.459.130.221.9w/ DenseCRF-

* training log can be found here.

Environments

Citations

Please consider citing our paper in your publications if the project helps your research.

@inproceedings{hsu2019bbtp,
  title     =  {Weakly Supervised Instance Segmentation using the Bounding Box Tightness Prior},
  author    =  {Cheng-Chun Hsu, Kuang-Jui Hsu, Chung-Chi Tsai, Yen-Yu Lin, Yung-Yu Chuang},
  booktitle =  {Neural Information Processing Systems},
  year      =  {2019}
}