Home

Awesome

FreeMask

This codebase provides the official PyTorch implementation of our NeurIPS 2023 paper:

FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models</br> Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, Hengshuang Zhao</br> In Conference on Neural Information Processing Systems (NeurIPS), 2023</br> [Paper] [Datasets] [Models] [Logs] [BibTeX]

TL;DR

We generate diverse synthetic images from semantic masks, and use these synthetic pairs to boost the fully-supervised semantic segmentation performance.

<p align="left"> <img src="./docs/vis.png" width=100% height=100% class="center"> </p>
<p align="left"> <img src="./docs/pipeline.png" width=100% height=100% class="center"> </p>

Results

ADE20K

ModelBackboneReal Images+ Synthetic ImagesGain ($\Delta$)Download
Mask2FormerSwin-T48.752.0+3.3ckpt | log
Mask2FormerSwin-S51.653.3+1.7ckpt | log
Mask2FormerSwin-B52.453.7+1.3ckpt | log
SegFormerMiT-B245.647.9+2.3ckpt | log
SegFormerMiT-B448.550.6+2.1ckpt | log
SegmenterViT-S46.247.9+1.7ckpt | log
SegmenterViT-B49.651.1+1.5ckpt | log

COCO-Stuff-164K

ModelBackboneReal Images+ Synthetic ImagesGain ($\Delta$)Download
Mask2FormerSwin-T44.546.4+1.9ckpt | log
Mask2FormerSwin-S46.847.6+0.8ckpt | log
SegFormerMiT-B243.544.2+0.7ckpt | log
SegFormerMiT-B445.846.6+0.8ckpt | log
SegmenterViT-S43.544.8+1.3ckpt | log
SegmenterViT-B46.047.5+1.5ckpt | log

High-Quality Synthetic Datasets

We share our already processed synthetic ADE20K and COCO-Stuff-164K datasets below. The ADE20K-Synthetic dataset is 20x larger than its real counterpart, while the COCO-Synthetic is 6x larger than its real counterpart.

Getting Started

Installation

Install MMSegmentation:

pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
pip install "mmsegmentation>=1.0.0"
pip install "mmdet>=3.0.0rc4"

Download Real Datasets

Follow the instructions to download the ADE20K and COCO-Stuff-164K real datasets. The COCO annotations need to be pre-processed following the instructions.

Download Synthetic Datasets

Please see above.

Note:

Usage

bash dist_train.sh <config> 8

Generate and Pre-process Synthetic Images (Optional)

We have provided the processed synthetic images above. You can directly use them to train a stronger segmentation model. However, if you want to generate additional images by yourself, we introduce the generation and pre-processing steps below.

Generate Synthetic Images

We strictly follow FreestyleNet for initial image generation. Please refer to their instructions. You can change the random seed to produce multiple synthetic images from a semantic mask.

Pre-process Synthetic Images

Our work focuses on this part.

Filter out Noisy Synthetic Regions

python preprocess/filter.py <config> <checkpoint> --real-img-path <> --real-mask-path <> --syn-img-path <> --syn-mask-path <> --filtered-mask-path <>

We use the pre-trained SegFormer-B4 model to calculate class-wise mean loss on real images and then filter out noisy synthetic regions.

Re-sample Synthetic Images based on Mask-level Hardness

python preprocess/resample.py --real-mask-path <> --syn-img-path <> --syn-mask-path <> --resampled-syn-img-path <> --resampled-syn-mask-path <> 

Acknowledgment

We thank FreestyleNet for providing their mask-to-image synthesis models.

Citation

If you find this project useful, please consider citing:

@inproceedings{freemask,
  title={FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models},
  author={Yang, Lihe and Xu, Xiaogang and Kang, Bingyi and Shi, Yinghuan and Zhao, Hengshuang},
  booktitle={NeurIPS},
  year={2023}
}