Awesome

Diverse Co-training

Official PyTorch implementation of ICCV 2023 paper "Diverse Cotraining Makes Strong Semi-Supervised Segmentor".

Abstract.

Deep co-training has been introduced to semi-supervised segmentation and achieves impressive results, yet few studies have explored the working mechanism behind it. In this work, we revisit the core assumption that supports co-training: multiple compatible and conditionally independent views. By theoretically deriving the generalization upper bound, we prove the prediction similarity between two models negatively impacts the model's generalization ability. However, most current co-training models are tightly coupled together and violate this assumption. Such coupling leads to the homogenization of networks and confirmation bias which consequently limits the performance. To this end, we explore different dimensions of co-training and systematically increase the diversity from the aspects of input domains, different augmentations and model architectures to counteract homogenization. Our \textit{Diverse Co-training} outperforms the state-of-the-art (SOTA) methods by a large margin across different evaluation protocols on the Pascal and Cityscapes. For example. we achieve the best mIoU of 76.2%, 77.7% and 80.2% on Pascal with only 92, 183 and 366 labeled images, surpassing the previous best results by more than 5%.

Results

Pascal

labeled data sampled from high-quality training set

Labeled images are sampled from the original high-quality training set. Results are obtained by DeepLabv3+ with ResNet-101 and SegFormer-b3.

labeled data sampled from blened training set

Labeled images are sampled from the blened training set. Results obtained by DeepLabv3+ with ResNet-50 and SegFormer-b2.

Results obtained by DeepLabv3+ with ResNet-101 and SegFormer-b3.

Cityscapes

Results are obtained by DeepLabv3+ with ResNet-50/101 and SegFormer-b2/b3 with resolution 769x769. Results of U²PL are from UniMatch.

Training Logs

We release the training logs in logs folder. Training logs on CItyscapes dataset can be found in logs/cityscapes. Training logs on VOC dataset can be found in logs/voc.

Getting Started

Installation

conda create -n cotraining python=3.7
conda activate cotraining
pip install torch torchvision torchaudio
pip install -r requirements.txt

We use implementation of SegFormer from mmsegmentation, so installation of mmcv and mmsegmentation libraries is needed

pip install openmim
mim install mmcv
pip install mmsegmentation

Proprosessing of DCT domain requires jpeg2dct and PyTurboJPEG libraries.

Before installing jpeg2dct, first install either libjpeg or libjpeg-turbo library.

apt install libjpeg-turbo
pip install jpeg2dct PyTurboJPEG

For more details regarding the installation of jpeg2dct, we refer to jpeg2dct. We also refer to DCTNet for more details of DCT transform.

Pretrained Backbone:

We provide the pretrain as followed:

ResNet-50 | ResNet-101 | ResNet-50-dct | ResNet-101-dct

├── ./pretrained
    ├── resnet50.pth
    ├── resnet101.pth
    ├── resnet50_dct.pth
    └── resnet101_dct.pth

Note: The ResNet variants all use official weights and we have pretrained DCT ResNet with comparable performance as ResNet couterparts. More details can be found in our paper.

Dataset:

• Pascal: JPEGImages | SegmentationClass
• Cityscapes: leftImg8bit | gtFine

├── [Your Pascal Path]
    ├── JPEGImages
    └── SegmentationClass
    
├── [Your Cityscapes Path]
    ├── leftImg8bit
    └── gtFine

Usage

Diverse Co-training

# use torch.distributed.launch
# To start training, the general format is as followed
sh <script> <num_gpu> <port> <data partition> <threshold>
# e.g. bash tools/voc/dist_train_cotraining_2cps.sh 4 29873 1_16 0.0

# we also provide a srun script for training on slurm cluster
# e.g. bash tools/voc/srun_train_cotraining_2cps.sh 4 29873 1_16 0.0

In order to run on different labeled data partitions or different datasets, please modify:

config, labeled_id_path, unlabeled_id_path, and save_path in the training shell script.

Supervised Baseline

Modify the py file to supervised.py in the script, and double the batch_size in the configuration file if you use the same number of GPUs as semi-supervised setting (no need to change lr).

If you want to run supervised on DCT input domain, follow the above instructions to modify supervised_dct.py in the script, everything is the same except for the input domain is changed.

Citation

If you find this project useful, please consider citing:

@InProceedings{Li_2023_ICCV,
    author    = {Li, Yijiang and Wang, Xinjiang and Yang, Lihe and Feng, Litong and Zhang, Wayne and Gao, Ying},
    title     = {Diverse Cotraining Makes Strong Semi-Supervised Segmentor},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2023},
    pages     = {16055-16067}
}