Awesome

[ICCV'23 Oral] Unmasking Anomalies in Road-Scene Segmentation

[arXiv]

https://github.com/shyam671/Mask2Anomaly-Unmasking-Anomalies-in-Road-Scene-Segmentation/assets/17329284/096effcd-51c8-4b1b-9b2b-f6746d4f6437

Run our demo using Colab:

Installation

Please follow the Installation Instruction to set up the codebase.

Datasets

We have three different sets of dataset used for training, ood-fine-tuning, and anomaly inference. Please follow the below steps to set-up each set.

Inlier Dataset(Cityscapes/Streethazard): consists of only inlier classes that can be prepared by following the same structure as given here.
Outlier Supervision Dataset(MS-COCO): helps fine-tune the model pre-trained on the inlier dataset on ood-objects. The outlier dataset is created by using this script and then changing the cfg.MODEL.MASK_FORMER.ANOMALY_FILEPATH accordingly.
Anomaly Dataset (validation): can be downloaded using this link. Please unzip the file and place it preferably in the dataset folder.

Training and Inference

The training of the model takes place in two stages:
- [Stage1] Training of inlier dataset performed using the command: Cityscapes Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file configs/cityscapes/semantic-segmentation/anomaly_train.yaml. Streethazard Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file 'configs/streethazard/streethazard_training.yaml'.
- [Stage2] We fine-tune the weights of the model using following command: Cityscapes Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file configs/cityscapes/semantic-segmentation/anomaly_ft.yaml. Streethazard Dataset: CUDA_VISIBLE_DEVICES=1 python train_net.py --num-gpus 1 --config-file 'configs/streethazard/streethazard_ft.yaml' .
During inference, we use the final-weights obtained after fine-tuning: CUDA_VISIBLE_DEVICES=1 python anomaly_utils/anomaly_inference.py --input '/path-to-images-files/*.jpg' --config-file '/path-to-anomaly-inference-config/anomaly_inference.yaml'
We provide all the commands for training, ood-fine-tune, and anomaly inference in run.sh and corresponding config files at /configs/cityscapes /semantic-segmentation/.
To perform anomaly segmentation using pre-trained models, download the model from shared Google Drive link and then change the model weight path in /configs/cityscapes/semantic-segmentation/anomaly_inference.yaml.
- Single Pretained model for SMIYC, Fishyscapes, LostAndFound, and Road Anomaly: Link
- StreetHazard: Link

Docker Image

We provide a singularity image similar to docker that provides anomaly output without needing library installation/GPUs.
Install Singularity following the instructions. Link
Download the .sif image from link
Run the command singularity run --bind {input-path-to-image-datset}:/input,{output-path-to-save-segmentation-maps}:/output mask2former.sif

License

Shield:

The majority of Mask2Anomaly is licensed under a MIT License.

However portions of the project are available under separate license terms: Mask2Former and Swin-Transformer-Semantic-Segmentation is licensed under the MIT license, Deformable-DETR is licensed under the Apache-2.0 License.

Acknowledgement

We thank the authors of the codebases mentioned below, which helped build the repository.