Home

Awesome

Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations

<div align="center">

Paper video slides poster BibTeX

</div>

This is the official PyTorch implementation of our work: "Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations" published at CVPR 2021.

In this paper, we present some novel approaches constraining the feature space for continual learning semantic segmentation models. The evaluation on Pascal VOC2012 and on ADE20K validated our method.

teaser <br/>

Requirements

This repository uses the following libraries:

We also assume to have installed pytorch.distributed package.

All the dependencies are listed in the requirements.txt file which can be used in conda as: <br/>conda create --name <env> --file requirements.txt

How to download data

In this project we use two dataset, ADE20K and Pascal-VOC 2012. We provide the scripts to download them in 'data/download_<dataset_name>.sh'. The script takes no inputs and you should use it in the target directory (where you want to download data).

How to perform training

The most important file is run.py, that is in charge to start the training or test procedure. To run it, simpy use the following command:

python -m torch.distributed.launch --nproc_per_node=<num_GPUs> run.py --data_root <data_folder> --name <exp_name> .. other args ..

The default is to use a pretraining for the backbone used, which is the one officially released by PyTorch models and it will be downloaded automatically. If you don't want to use pretrained, please use --no-pretrained.

There are many options and you can see them all by using --help option. Some of them are discussed in the following:

For all the details please follow the information provided using the help option.

Example training commands

We provide some example scripts in the *.slurm and *.bat files. <br/> For instance, to run the step 0 of 19-1 VOC2012 you can run:<br/>

python -u -m torch.distributed.launch 1> 'outputs/19-1/output_19-1_step0.txt' 2>&1 \
--nproc_per_node=1 run.py \
--batch_size 8 \
--logdir logs/19-1/ \
--dataset voc \
--name FT \
--task 19-1 \
--step 0 \
--lr 0.001 \
--epochs 30 \
--debug \
--sample_num 10 \
--unce \
--loss_de_prototypes 1 \
--where_to_sim GPU_windows

Note: loss_de_prototypes is set to 1 only for having the prototypes computed in the 0-th step (no distillation is actually computed of course).

Then, the step 1 of the same scenario can be computed simply as:

python -u -m torch.distributed.launch 1> 'outputs/19-1/output_19-1_step1.txt'  2>&1 \
--nproc_per_node=1 run.py \
--batch_size 8 \
--logdir logs/19-1/ \
--dataset voc \
--task 19-1 \
--step 1 \
--lr 0.0001 \
--epochs 30 \
--debug \
--sample_num 10 \
--where_to_sim GPU_windows \
--method SDR \
--step_ckpt 'logs/19-1/19-1-voc_FT/19-1-voc_FT_0.pth'

The results obtained are reported inside the outputs/ and logs/ folder, which can be downloaded here, and are 0.4% of mIoU higher than those reported in the main paper due to a slightly changed hyperparameter.

To run other approaches it is sufficient to change the --method parameter into one of the following: FT, LWF, LWF-MC, ILT, EWC, RW, PI, MIB, CIL, SDR.

Note: for the best results, the hyperparameters may change. Please see further details on the hyperparameters section below.

Once you trained the model, you can see the result on tensorboard (we perform the test after the whole training) or on the output files. or you can test it by using the same script and parameters but using the option

--test

that will skip all the training procedure and test the model on test data.

Do you want to try our constraints on your codebase or task?

If you want to try our novel constraints on your codebase or on a different problem you can check the utils/loss.py file. Here, you can take the definitions of the different losses and embed them into your codebase <br/> The names of the variables could be interpreted as:<br/>

Range for the Hyper-parameters

For what concerns the hyperparameters of our approach:

So, writing explicitly all the parameters, the command would look something like the following:

python -u -m torch.distributed.launch 1> 'outputs/19-1/output_19-1_step1_custom.txt'  2>&1 \
--nproc_per_node=1 run.py \
--batch_size 8 \
--logdir logs/19-1/ \
--dataset voc \
--task 19-1 \
--step 1 \
--lr 0.0001 \
--epochs 30 \
--debug \
--sample_num 10 \
--where_to_sim GPU_windows \
--unce \
--loss_featspars $loss_featspars \
--lfs_normalization $lfs_normalization \
--lfs_shrinkingfn $lfs_shrinkingfn \
--lfs_loss_fn_touse $lfs_loss_fn_touse \
--loss_de_prototypes $loss_de_prototypes \
--loss_de_prototypes_sumafter \
--lfc_sep_clust $lfc_sep_clust \
--loss_fc $loss_fc \
--loss_kd $loss_kd \
--step_ckpt 'logs/19-1/19-1-voc_FT/19-1-voc_FT_0.pth'

<a name="Citation"></a>

Cite us

If you use this repository, please consider to cite

   @inProceedings{michieli2021continual,
   author = {Michieli, Umberto and Zanuttigh, Pietro},
   title  = {Continual Semantic Segmentation via Repulsion-Attraction of Sparse and Disentangled Latent Representations},
   booktitle = {Computer Vision and Pattern Recognition (CVPR)},
   year      = {2021},
   month     = {June}
   }

And our previous works ILT and its journal extension.

Acknowledgements

We gratefully acknowledge the authors of MiB paper for the insightful discussion and for providing the open source codebase, which has been the starting point for our work. <br/> We also acknowledge the authors of CIL for providing their code even before the official release.