Awesome
RCIL
[CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation<br> Chang-Bin Zhang<sup>1</sup>, Jia-Wen Xiao<sup>1</sup>, Xialei Liu<sup>1</sup>, Ying-Cong Chen<sup>2</sup>, Ming-Ming Cheng<sup>1</sup><br> <sup>1</sup> <sub>College of Computer Science, Nankai University</sub><br /> <sup>2</sup> <sub>The Hong Kong University of Science and Technology</sub><br />
News
- Our work can be reproduced with high version pytorch after fixing the issue caused by InplaceABNSync. Please refer to our modification. πππThanks to Zhengyuan and Jiawen.
- Our another work [EWF] has been accepted to CVPR 2023.
- Our method RCIL has been re-implemented in the framework π₯π₯π₯ [CSSegmentation].
Method
<img width="1230" alt="ζͺε±2022-04-09 δΈε1 02 44" src="https://user-images.githubusercontent.com/35215543/162488465-73c56e73-8d5b-4406-941f-85497673c419.png">Update
Support CIL on cityscapes dataset- init code for Classification
We have fixed bugs in the repositoryAdd training scripts for ADE20K09/04/2022 init code for segmentation09/04/2022 init readme
Benchmark and Setting
There are two commonly used settings, disjoint
and overlapped
.
In the disjoint
setting, assuming we know all classes in the future, the images in the current training step do not contain any classes in the future. The overlapped
setting allows potential classes in the future to appear in the current training images. We call each training on the newly added dataset as a step. Formally, X-Y
denotes the continual setting in our experiments, where X
denotes the number of classes that we need to train in the first step. In each subsequent learning step, the newly added dataset contains Y
classes.
There are some settings reported in our paper. You can also try it on other any custom settings.
-
Continual Class Segmentation:
- PASCAL VOC 2012 dataset:
- 15-5 overlapped
- 15-5 disjoint
- 15-1 overlapped
- 15-1 disjoint
- 10-1 overlapped
- 10-1 disjoint
- ADE20K dataset:
- 100-50 overlapped
- 100-10 overlapped
- 50-50 overlapped
- 100-5 overlapped
- PASCAL VOC 2012 dataset:
-
Continual Domain Segmentation:
- Cityscapes:
- 11-5
- 11-1
- 1-1
- Cityscapes:
-
Extension Experiments on Continual Classification
- ImageNet-100
- 50-10
- ImageNet-100
Performance
- Continual Class Segmentation on PASCAL VOC 2012
Method | Pub. | 15-5 disjoint | 15-5 overlapped | 15-1 disjoint | 15-1 overlapped | 10-1 disjoint | 10-1 overlapped | 5-3 overlapped | 5-3 disjoint |
---|---|---|---|---|---|---|---|---|---|
LWF | TPAMI 2017 | 54.9 | 55.0 | 5.3 | 5.5 | 4.3 | 4.8 | ||
ILT | ICCVW 2019 | 58.9 | 61.3 | 7.9 | 9.2 | 5.4 | 5.5 | ||
MiB | CVPR 2020 | 65.9 | 70.0 | 39.9 | 32.2 | 6.9 | 20.1 | ||
SDR | CVPR 2021 | 67.3 | 70.1 | 48.7 | 39.5 | 14.3 | 25.1 | ||
PLOP | CVPR 2021 | 64.3 | 70.1 | 46.5 | 54.6 | 8.4 | 30.5 | ||
Ours | CVPR 2022 | 67.3 | 72.4 | 54.7 | 59.4 | 18.2 | 34.3 | 42.88 |
- Continual Class Segmentation on ADE20K
Method | Pub. | 100-50 overlapped | 100-10 overlapped | 50-50 overlapped | 100-5 overlapped |
---|---|---|---|---|---|
ILT | ICCVW 2019 | 17.0 | 1.1 | 9.7 | 0.5 |
MiB | CVPR 2020 | 32.8 | 29.2 | 29.3 | 25.9 |
PLOP | CVPR 2021 | 32.9 | 31.6 | 30.4 | 28.7 |
Ours | CVPR 2022 | 34.5 | 32.1 | 32.5 | 29.6 |
- Continual Domain Segmentation on Cityscapes
Method | Pub. | 11-5 | 11-1 | 1-1 |
---|---|---|---|---|
LWF | TPAMI 2017 | 59.7 | 57.3 | 33.0 |
LWF-MC | CVPR 2017 | 58.7 | 57.0 | 31.4 |
ILT | ICCVW 2019 | 59.1 | 57.8 | 30.1 |
MiB | CVPR 2020 | 61.5 | 60.0 | 42.2 |
PLOP | CVPR 2021 | 63.5 | 62.1 | 45.2 |
Ours | CVPR 2022 | 64.3 | 63.0 | 48.9 |
Dataset Prepare
- PASCVAL VOC 2012
sh data/download_voc.sh
- ADE20K
sh data/download_ade.sh
- Cityscapes
sh data/download_cityscapes.sh
Environment
conda install --yes --file requirements.txt
(Higher version pytorch should be suitable.)- Install inplace-abn
Training
- Dowload pretrained model from ResNet-101_iabn to
pretrained/
- We have prepared some training scripts in
scripts/
. You can train the model by
sh scripts/voc/rcil_10-1-overlap.sh
Inference
You can simply modify the bash file by adding --test
, like
CUDA_VISIBLE_DEVICES=${GPU} python3 -m torch.distributed.launch --master_port ${PORT} --nproc_per_node=${NB_GPU} run.py --data xxx ... --test
Reference
If this work is useful for you, please cite us by:
@inproceedings{zhang2022representation,
title={Representation Compensation Networks for Continual Semantic Segmentation},
author={Zhang, Chang-Bin and Xiao, Jia-Wen and Liu, Xialei and Chen, Ying-Cong and Cheng, Ming-Ming},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={7053--7064},
year={2022}
}
Contact
If you have any questions about this work, please feel easy to contact us (zhangchbin ^ mail.nankai.edu.cn or zhangchbin ^ gmail.com).
Thanks
This code is heavily borrowed from [MiB] and [PLOP].
Awesome Continual Segmentation
There is a collection of AWESOME things about continual semantic segmentation, including papers, code, demos, etc. Feel free to pull request and star.
2022
- Representation Compensation Networks for Continual Semantic Segmentation [CVPR 2022] [PyTorch]
- Self-training for Class-incremental Semantic Segmentation [TNNLS 2022] [PyTorch]
- Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation [TPAMI 2022] [[PyTorch]]
2021
- PLOP: Learning without Forgetting for Continual Semantic Segmentation [CVPR 2021] [PyTorch]
- Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations [CVPR2021] [PyTorch]
- An EM Framework for Online Incremental Learning of Semantic Segmentation [ACM MM 2021] [PyTorch]
- SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning [NeurIPS 2021] [PyTorch]
2020
- Modeling the Background for Incremental Learning in Semantic Segmentation [CVPR 2020] [PyTorch]
2019
- Incremental Learning Techniques for Semantic Segmentation [ICCV Workshop 2019] [PyTorch]