Awesome
CTRL-UDA
This is the official PyTorch implementation of our CVPR'21 paper "Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation" [arXiv] [CVF] <img src="./extra/overview.png" alt="alt text" width="100%" height="100%">
Setting up a virtual environment
The following shell commands create a new python virtual environment, install the dependencies, and activate it:
python -m venv ~/venv_ctrl_uda
source ~/venv_ctrl_uda/bin/activate
pip install -r requirements.txt
Train/Test a CTRL-UDA model
# STEP-1: First train CTRL-UDA
main.py --expid=<experiment id> --reso='<"LOW" or "FULL">' --exp_root_dir=<output folder path> --data_root=<dataset root folder path> --pret_model=<pretrained model path>
# STEP-2: Next, train CTRL-UDA with ISL
main.py --expid=<experiment id> --reso='<"LOW" or "FULL">' --exp_root_dir=<output folder path> --data_root=<dataset root folder path> --model_path=<trained model path in STEP-1>
Where experiment ids are:
Exp. Id | Description |
---|---|
1 | SYNTHIA to Cityscapes 16 classes |
2 | SYNTHIA to Cityscapes 7 classes (full resolution) |
3 | SYNTHIA to Cityscapes 7 classes (low resolution) |
4 | SYNTHIA to Mapillary 7 classes (full resolution) |
5 | SYNTHIA to Mapillary 7 classes (low resolution) |
Download the weights used to initialize the ResNet-101 backbone from Google Drive. and set the path to --pret_model in STEP-1.
Experiments
Datasets
-
SYNTHIA: Please follow the instructions here to download the images. We used the SYNTHIA-RAND-CITYSCAPES (CVPR16) split. Download the segmentation labels here here. Please follow the dataset directory structure:
<data_root>/Synthia/RAND_CITYSCAPES % SYNTHIA dataset root ├── RGB ├── parsed_LABELS └── Depth
-
Cityscapes: Please follow the instructions in Cityscape to download the images and validation ground-truths. Please follow the dataset directory structure:
<data_root>/Cityscapes % Cityscapes dataset root ├── leftImg8bit │ ├── train │ └── val └── gtFine └── val
-
Mapillary: Please follow the instructions in Mapillary to download the images and validation ground-truths. Please follow the dataset directory structure:
<data_root>/Mapillary-Vista % Mapillary-Vista dataset root ├── train │ └── images └── validation ├── images └── labels
Reported Results
Reported Results on SYNTHIA ⟶ Cityscapes (16 classes) (Exp. Id: 1)
Method | mIoU*(13) | mIoU(16) |
---|---|---|
SPIGAN-no-PI [34] | 41.2 | 35.8 |
SPIGAN [34] | 42.4 | 36.8 |
AdaptSegnet [55] | 45.9 | -- |
AdaptPatch [56] | 46.3 | -- |
CLAN [41] | 47.8 | -- |
Advent [61] | 47.6 | 40.8 |
DADA [62] | 49.8 | 42.6 |
Ours (best mIoU) | 51.3 | 44.9 |
Ours (confidence) | 51.0 ± 0.7 | 44.3 ± 0.6 |
Reported Results on SYNTHIA ⟶ Cityscapes (7 classes full resolution) (Exp. Id: 2)
Method | mIoU |
---|---|
Advent [61] | 67.2 |
DADA [62] | 69.2 |
Ours (best mIoU) | 72.3 |
Ours (confidence) | 71.3 ± 0.9 |
Reported Results on SYNTHIA ⟶ Cityscapes (7 classes low resolution) (Exp. Id: 3)
Method | mIoU |
---|---|
SPIGAN-no-PI [34] | 46.0 |
SPIGAN [34] | 53.4 |
Advent [61] | 59.4 |
DADA [62] | 63.4 |
Ours (best mIoU) | 65.4 |
Ours (confidence) | 64.9 ± 0.3 |
Reported Results on SYNTHIA ⟶ Mapillary (7 classes full resolution) (Exp. Id: 4)
Method | mIoU |
---|---|
Advent [61] | 65.2 |
DADA [62] | 67.6 |
Ours (best mIoU) | 71.8 |
Ours (confidence) | 70.9 ± 0.7 |
Reported Results on SYNTHIA ⟶ Mapillary (7 classes low resolution) (Exp. Id: 4)
Method | mIoU |
---|---|
SPIGAN-no-PI [34] | 26.8 |
SPIGAN [34] | 44.1 |
Advent [61] | 54.0 |
DADA [62] | 55.8 |
Ours (best mIoU) | 61.9 |
Ours (confidence) | 61.3 ± 0.5 |
Pre-trained models
Pre-trained models are provided here Google Drive.
Reproducibility
For ensuring reproducibility, we reported the confidence intervals of the class-specific IOUs and mIoUs on the five different UDA settings (Table 1 and 2 in the main paper). For each setting, we trained our model five times with different random seed values and reported the mean and standard deviation (of the mIoU) over these five runs. Due to the randomness involved in deep generative models, the same seed value might yield slightly different mIoU. In that case, you may need to train a few times or train a bit longer (by changing the EARLY_STOP) to reach a comparable performance.
Citation
Please cite our work if you find it useful.
@InProceedings{Saha_2021_CVPR,
author = {Saha, Suman and Obukhov, Anton and Paudel, Danda Pani and Kanakis, Menelaos and Chen, Yuhua and Georgoulis, Stamatios and Van Gool, Luc},
title = {Learning To Relate Depth and Semantics for Unsupervised Domain Adaptation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2021},
pages = {8197-8207}
}
Acknowledgement
- DADA is used as our baseline and coding reference official.