Home

Awesome

UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation [NeurIPS 2024]

:rocket: Updates

Abstract

Image segmentation is a crucial vision task that groups pixels within an image into semantically meaningful segments, which is pivotal in obtaining a fine-grained understanding of real-world scenes. However, an increasing privacy concern exists regarding training large-scale image segmentation models on unauthorized private data. In this work, we exploit the concept of unlearnable examples to make images unusable to model training by generating and adding unlearnable noise into the original images. Particularly, we propose a novel Unlearnable Segmentation (UnSeg) framework to train a universal unlearnable noise generator that is capable of transforming any downstream images into their unlearnable version. The unlearnable noise generator is finetuned from the Segment Anything Model (SAM) via bilevel optimization on an interactive segmentation dataset towards minimizing the training error of a surrogate model that shares the same architecture with SAM but is trained from scratch. We empirically verify the effectiveness of UnSeg across 6 mainstream image segmentation tasks, 10 widely used datasets, and 7 different network architectures, and show that the unlearnable images can reduce the segmentation performance by a large margin. Our work provides useful insights into how to leverage foundation models in a data-efficient and computationally affordable manner to protect images against image segmentation models.

<p align="center"> <img src="UnSeg.png" width="700"/> </p>

1. Installation

Clone this repository and navigate to project folder

git clone https://github.com/sunye23/UnSeg.git
cd UnSeg

Quick Installation

conda create --name unseg python=3.8 -y
conda activate unseg
bash install.sh
git clone https://github.com/facebookresearch/detectron2.git
cd detectron2
pip install -e .
pip install git+https://github.com/cocodataset/panopticapi.git
pip install git+https://github.com/mcordts/cityscapesScripts.git

We organize the training folder as follows.

Project
|____data
| |____cascade_psp
| |____DIS5K
| |____thin_object_detection
|____eval
|____scripts
|____generate
|____weight_dir
| |____Generator_unseg_final.pth
|____pretrained_checkpoint
| |____modified_mae_pretrain_vit_base.pth
| |____sam_vit_b_backbone.pth
| |____sam_vit_b_maskdecoder.pth
| |____sam_vit_b_prompt_decoder.pth
|____train.py
|____utils
|____segment_anything_training
|____work_dirs

2. Training Data Preparation

Our training data is based on HQSeg-44K and can be downloaded from hugging face link

Expected dataset structure for HQSeg-44K

data
|____DIS5K
|____cascade_psp
| |____DUTS-TE
| |____DUTS-TR
| |____ecssd
| |____fss_all
| |____MSRA_10K
|____thin_object_detection
| |____COIFT
| |____HRSOD
| |____ThinObject5K

3. Pretrained checkpoint Preparation

Download the model weight here and put them into right dir.

4. Training

Model training is based on RTX3090 x 8.

bash scripts/run_train.sh

5. UEs Generation

UEs generation is based on RTX3090 x 8.

Before generating ues, place the clean images and the corresponding groundtruths in the correct directory. For the datasets such as coco, cityscapes, and ade20k, please refer to Mask2former for data preparation.

🔥TIPs: UnSeg can also generate UEs for other image segmentation datasets easily. Simply modify the code in the generation directory, place the images and their corresponding object masks in the correct paths, and then run the scripts.

Here we only provide the download guidance for the pascal voc dataset.

<details> <summary> PASCAL VOC 2012 </summary>

For Pascal VOC dataset.

bash scripts/generate_pascal.sh

For Ade20K dataset.

bash scripts/generate_ade20k.sh

For COCO dataset.

bash scripts/generate_coco.sh

For Cityscapes dataset.

bash scripts/generate_cityscapes.sh

For interactive dataset.

bash scripts/generate_interactive.sh

For remote sensing dataset, please refer to Rsprompter for data preparation.

bash scripts/generate_remote.sh

For medical image dataset.

bash scripts/generate_medical.sh

6. Evaluation

For the DeepLabv1 evaluation, we mainly employ the code provided by MCTFormer. You should follow the work to prepare the weights and environments. We also provide the code and the script under the eval dir. You could use the command below:

bash eval/run_seg.sh

For mainstream image segmentation evaluation, please refer to Mask2former.

For interactive image segmentation evaluation, please refer to SAM-HQ.

For remote sensing evaluation, please refer to Rsprompter.

For medical image segmentation evaluation, our code is based on the segmentation_models_pytorch codebase.

For object detection evaluation, please refer to DINO.

Citation

If you find UnSeg useful for your research and applications, please kindly cite using this BibTeX:

@inproceedings{sun2024unseg,
  title={UnSeg: One Universal Unlearnable Example Generator is Enough against All Image Segmentation},
  author={Sun, Ye and Zhang, Hao and Zhang, Tiehua and Ma, Xingjun and Jiang, Yu-Gang},
  booktitle={NeurIPS},
  year={2024}
}