Home

Awesome

[CVPR 2024] Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation

zhixiang wei<sup>1</sup>, lin chen<sup>2</sup>, et al. <br /> <sup>1</sup> University of Science of Techonology of China <sup>2</sup> Shanghai AI Laboratory

PWC <br /> PWC <br /> PWC <br /> Project page: https://zxwei.site/rein

Paper: https://arxiv.org/pdf/2312.04265.pdf

Rein is a efficient and robust fine-tuning method, specifically developed to effectively utilize Vision Foundation Models (VFMs) for Domain Generalized Semantic Segmentation (DGSS). It achieves SOTA on Cityscapes to ACDC, and GTAV to Cityscapes+Mapillary+BDD100K. Using only synthetic data, Rein achieved an mIoU of 78.4% on Cityscapes validation set! Using only the data from the Cityscapes training set, we achieved an average mIoU of 77.6% on ACDC test set! Rein Framework

Visualization

Trained on Cityscapes, Rein generalizes to unseen driving scenes and cities: Nighttime Shanghai, Foggy Countryside, and Rainy Hollywood.

![night]

![rain]

![fog]

Performance Under Various Settings (DINOv2).

SettingmIoUConfigLog & Checkpoint
GTAV $\rightarrow$ Cityscapes66.7configlog & checkpoint
+Synthia $\rightarrow$ Cityscapes72.2configlog & checkpoint
+UrbanSyn $\rightarrow$ Cityscapes78.4configlog & checkpoint
+1/16 of Cityscapes training $\rightarrow$ Cityscapes82.5configlog & checkpoint
GTAV $\rightarrow$ BDD100K60.0configlog & checkpoint
Cityscapes $\rightarrow$ ACDC77.6configlog & checkpoint
Cityscapes $\rightarrow$ Cityscapes-C60.0configlog & checkpoint

Performance For Various Backbones (Trained on GTAV).

SettingPretrainingCitys. mIoUConfigLog & Checkpoint
ResNet50ImageNet1k49.1configlog & checkpoint
ResNet101ImageNet1k45.9configlog & checkpoint
ConvNeXt-LargeImageNet21k57.9configlog & checkpoint
ViT-SmallDINOv255.3configlog & checkpoint
ViT-BaseDINOv264.3configlog & checkpoint
CLIP-LargeOPENAI58.1configlog & checkpoint
SAM-HugeSAM59.2configlog & checkpoint
EVA02-LargeEVA0267.8configlog & checkpoint

Citation

If you find our code or data helpful, please cite our paper:

@InProceedings{Wei_2024_CVPR,
    author    = {Wei, Zhixiang and Chen, Lin and Jin, Yi and Ma, Xiaoxiao and Liu, Tianle and Ling, Pengyang and Wang, Ben and Chen, Huaian and Zheng, Jinjin},
    title     = {Stronger Fewer \& Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentation},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {28619-28630}
}

šŸ”„ News!

Try and Test

Experience the demo: Users can open demo.ipynb in any Jupyter-supported editor to explore our demonstration. Demo Preview

For testing on the cityscapes dataset, refer to the 'Install' and 'Setup' sections below.

Environment Setup

To set up your environment, execute the following commands:

conda create -n rein -y
conda activate rein
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia -y
pip install -U openmim
mim install mmengine
mim install "mmcv>=2.0.0"
pip install "mmsegmentation>=1.0.0"
pip install "mmdet>=3.0.0"
pip install xformers=='0.0.20' # optional for DINOv2
pip install -r requirements.txt
pip install future tensorboard

Dataset Preparation

The Preparation is similar as DDB.

Cityscapes: Download leftImg8bit_trainvaltest.zip and gt_trainvaltest.zip from Cityscapes Dataset and extract them to data/cityscapes.

Mapillary: Download MAPILLARY v1.2 from Mapillary Research and extract it to data/mapillary.

GTA: Download all image and label packages from TU Darmstadt and extract them to data/gta.

Prepare datasets with these commands:

cd Rein
mkdir data
# Convert data for validation if preparing for the first time
python tools/convert_datasets/gta.py data/gta # Source domain
python tools/convert_datasets/cityscapes.py data/cityscapes
# Convert Mapillary to Cityscapes format and resize for validation
python tools/convert_datasets/mapillary2cityscape.py data/mapillary data/mapillary/cityscapes_trainIdLabel --train_id
python tools/convert_datasets/mapillary_resize.py data/mapillary/validation/images data/mapillary/cityscapes_trainIdLabel/val/label data/mapillary/half/val_img data/mapillary/half/val_label

(Optional) ACDC: Download all image and label packages from ACDC and extract them to data/acdc.

(Optional) UrbanSyn: Download all image and label packages from UrbanSyn and extract them to data/urbansyn.

The final folder structure should look like this:

Rein
ā”œā”€ā”€ ...
ā”œā”€ā”€ checkpoints
ā”‚   ā”œā”€ā”€ dinov2_vitl14_pretrain.pth
ā”‚   ā”œā”€ā”€ dinov2_rein_and_head.pth
ā”œā”€ā”€ data
ā”‚   ā”œā”€ā”€ cityscapes
ā”‚   ā”‚   ā”œā”€ā”€ leftImg8bit
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ train
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ val
ā”‚   ā”‚   ā”œā”€ā”€ gtFine
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ train
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ val
ā”‚   ā”œā”€ā”€ bdd100k
ā”‚   ā”‚   ā”œā”€ā”€ images
ā”‚   ā”‚   |   ā”œā”€ā”€ 10k
ā”‚   ā”‚   ā”‚   |    ā”œā”€ā”€ train
ā”‚   ā”‚   ā”‚   |    ā”œā”€ā”€ val
ā”‚   ā”‚   ā”œā”€ā”€ labels
ā”‚   ā”‚   |   ā”œā”€ā”€ sem_seg
ā”‚   ā”‚   |   |    ā”œā”€ā”€ masks
ā”‚   ā”‚   ā”‚   |    |    ā”œā”€ā”€ train
ā”‚   ā”‚   ā”‚   |    |    ā”œā”€ā”€ val
ā”‚   ā”œā”€ā”€ mapillary
ā”‚   ā”‚   ā”œā”€ā”€ training
ā”‚   ā”‚   ā”œā”€ā”€ cityscapes_trainIdLabel
ā”‚   ā”‚   ā”œā”€ā”€ half
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ val_img
ā”‚   ā”‚   ā”‚   ā”œā”€ā”€ val_label
ā”‚   ā”œā”€ā”€ gta
ā”‚   ā”‚   ā”œā”€ā”€ images
ā”‚   ā”‚   ā”œā”€ā”€ labels
ā”œā”€ā”€ ...

Pretraining Weights

Evaluation

Run the evaluation:

python tools/test.py configs/dinov2/rein_dinov2_mask2former_512x512_bs1x4.py checkpoints/dinov2_rein_and_head.pth --backbone dinov2_converted.pth

For most of provided release checkpoints, you can run this command to evluate

python tools/test.py /path/to/cfg /path/to/checkpoint --backbone /path/to/dinov2_converted.pth #(or dinov2_converted_1024x1024.pth)

Training

Start training in single GPU:

python tools/train.py configs/dinov2/rein_dinov2_mask2former_512x512_bs1x4.py

Start training in multiple GPU:

PORT=12345 CUDA_VISIBLE_DEVICES=1,2,3,4 bash tools/dist_train.sh configs/dinov2/rein_dinov2_mask2former_1024x1024_bs4x2.py NUM_GPUS

Generate full weights

Because we only fine-tune and save the REIN and head weights, if you need a complete set of segmentor weights, you need to use this script:

python generate_full_weights.py --segmentor_save_path SEGMENTOR_SAVE_PATH --backbone CONVERTED_BACKBONE --rein_head REIN_HEAD

FAQs

Acknowledgment

Our implementation is mainly based on following repositories. Thanks for their authors.

Star History

Star History Chart