Home

Awesome

Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport

Wentong Li, Yuqian Yuan, Song Wang, Jianke Zhu, Jianshu Li, Jian Liu, and Lei Zhang

Paper (arXiv). ICCV2023.

<img src="./data/point2mask.jpg" width="800px">

Environment Setup

conda create -n point2mask python=3.8 -y
conda activate point2mask
pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install openmim
mim install mmdet==2.17.0
mim install mmcv-full==1.3.9
git clone https://github.com/LiWentomng/Point2Mask.git
cd Point2Mask
pip install -r requirements.txt
sh setup.sh

Model Zoo

1.Single-point Supervision(P1)

Pascal VOC

BackboneSupervisionModelsPQPQ_thPQ_st
ResNet-50P1model53.751.990.5
ResNet-101P1model54.853.090.4
Swin-LP1model61.059.493.0

COCO

BackboneSupervisionModelsPQPQ_thPQ_st
ResNet-50P1model32.432.632.2
ResNet-101P1model34.034.333.5
Swin-LP1model37.037.036.9

Pascal VOC with COCO P1-pretrained model.

BackboneSupervisionModelsPQPQ_thPQ_st
ResNet-50P1model60.759.191.8
ResNet-101P1model63.261.892.3
Swin-LP1model64.262.793.2

2.Ten-point Supervision(P10)

Pascal VOC

BackboneSupervisionModelsPQPQ_thPQ_st
ResNet-50P10model59.157.591.8
ResNet-101P10model60.258.692.1

COCO

BackboneSupervisionModelsPQPQ_thPQ_st
ResNet-50P10model32.432.632.2
ResNet-101P10model36.737.335.7

Get Started

We use Pascal VOC and COCO datasets, please see Preparing datasets for Point2Mask.

Demo

To test our model with an input image, please run demo.py

python  demo.py --config-file ./configs/point2mask/voc/point2mask_voc_wsup_r50.py --weights /path/to/coco_r50.pth --input image.jpg --out-file prediction.jpg

Training

For VOC training:

CUDA_VISIBLE_DEVICE=0,1,2,3  tools/dist_train.sh configs/point2mask/voc/point2mask_voc_wsup_r50.py 4

For COCO training:

CUDA_VISIBLE_DEVICE=0,1,2,3,4,5,6,7  tools/dist_train.sh configs/point2mask/coco/point2mask_coco_wsup_r50.py 8

Note: our models for Pascal VOC are trained with 3090/V100 gpus, and for COCO with A100 GPUs. The Structured Edge (SE) model used for low-level edege is here.

Test

For PQ evaluation:

CUDA_VISIBLE_DEVICE=0,1,2,3  tools/dist_test.sh configs/point2mask/voc/point2mask_voc_wsup_r50.py work_dirs/xxx.pth 4  --eval pq

For visual results:

CUDA_VISIBLE_DEVICE=0 tools/dist_test.sh configs/point2mask/voc/point2mask_voc_wsup_r50.py work_dirs/xxx.pth 1 --show-dir xxx

Visual Examples

Visual Results on COCO with ResNet-101. <img src="./data/visualization.jpg" width="800px">

Visualization of the learning high-level boundary map. <img src="./data/highlevel_bundary_map.png" width="800px">

Acknowledgement

Code is largely based on PSPS, Panoptic Segformer, MMdetection.
Thanks for their great open-source projects!

Citation

@inproceedings{point2mask,
  title={Point2Mask: Point-supervised Panoptic Segmentation via Optimal Transport},
  author={Wentong Li, Yuqian Yuan, Song Wang, Jianke Zhu, Jianshu Li, Jian Liu and Lei Zhang},
  booktitle={Proceedinngs of IEEE International Conference on Computer Vision},
  year={2023}
}