Awesome
Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation
Project page | Paper | Video
Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation <br> Yingda Xia*, Yi Zhang*, Fengze Liu, Wei Shen, and Alan Yuille.<br> In ECCV 2020 (Oral).
Installation
Clone this repo.
git clone https://github.com/YingdaXia/SynthCP.git
cd SynthCP
This code has been tested with PyTorch 1.2.0 and python 3.7. Please install dependencies by
pip install -r requirements.txt
Dataset Preparation
Download Cityscapes dataset and place it under datasets/
.
Download StreetHards dataset following this repo. Please download both train and test data, and arrange them as anomaly/data/train
and anomaly/data/test
respectively.
Running the code on the Cityscpaes
First, download our checkpoint models from here and name the folder as `checkpoints/'. You should have a structure like this,
checkpoints/
|----fcn8s/
|----deeplab/
|----iounet/
|----cityscapes_label_only_c19/
|----caos/
|----caos-segmentation/
Obtaining paper results
Take FCN-8s as example, you will need to test the FCN model on the testset of Cityscapes. See scripts/eval_segmentation_models.sh
.
bash scripts/eval_segmentation_models.sh
Then run the scripts we provided to obtain numbers.
- SynthCP:
scripts/reproduce_synthcp_{fcn,deeplab}.sh
- Direct Prediction:
scripts/reproduce_synthcp_{fcn,deeplab}.sh
- MSP:
scripts/reproduce_synthcp_{fcn,deeplab}.sh
- MCDropout:
scripts/reproduce_mcdropout_{fcn,deeplab}.sh
- TCP:
scripts/reproduce_tcp.sh
Training
To train the synthesize module (SPADE), use the following script:
cd spade-cityscapes
bash run.sh
We provided our pre-trained SPADE (on the label-image pairs) in checkpoints/cityscapes_label_only_c19/
.
To train the comparison module, you need to first train the segmentation models using cross-validation on Cityscapes training set,
bash scripts/train_segmentation_models.sh
Then evaluate the model on the validation set of each fold (See the commented lines in scripts/eval_segmentation_models.sh
).
bash scripts/eval_segmentation_models.sh
Also run the synthesize module on the training set (See the commented lines in scripts/reproduce_synthcp_{fcn,deeplab}.sh
). Train the comparison module,
bash scripts/train_iounet.sh 0 $EXP_PATH $IOUNET_NAME $REC_PATH
Similar process for Direct Prediction, image-level MCDropout and TCP.
Running the code on StreetHazards dataset
First, train and test segmentation model to obtain segmentation predictions (saved in anomaly/data/test_result
by default).
cd anomaly
python train.py
python test.py
We provided our trained segmentation model in checkpoints/caos-segmentation
.
Then, train the synthesize module (SPADE).
cd ../spade-caos
bash run.sh
And use it to obtain reconstructions of the segmentation predictions (saved in anomaly/data/test_recon
by default).
bash eval_spade.sh
We also provided our trained GAN model in checkpoints/caos
. If you want to use it, please copy it to spade-caos/checkpoints/caos
.
Finally, segment anomaly objects by computing a feature-space distance between the images and the reconstructions.
cd ../anomaly
python eval_ood_rec.py
Citation
If you use this code for your research, please cite our papers.
@inproceedings{xia2020synthesize,
title={Synthesize then Compare: Detecting Failures and Anomalies for Semantic Segmentation},
author={Xia, Yingda and Zhang, Yi and Liu, Fengze and Shen, Wei and Yuille, Alan},
booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
year={2020}
}
Acknowledgments
The code for training and testing the synthesize module is extended from SPADE (Copyright (C) 2019 NVIDIA Corporation). The code for anomaly segmentation is extended from anomaly-seg, which is also built on semantic-segmentation-pytorch.