Awesome

Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer

²Alibaba Group 

The code repository for "Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer" in PyTorch.

News

[02/2023]🎉 Our paper has been accepted by CVPR2024.

[03/2023] 🌟 arXiv paper has been released.

[04/2024] 🌟 The code repository has been released.

Abstract

Class-incremental learning (CIL) aims to enable models to continuously learn new classes while overcoming catastrophic forgetting. The introduction of pre-trained models has brought new tuning paradigms to CIL. In this paper, we revisit different parameter-efficient tuning (PET) methods within the context of continual learning. We observe that adapter tuning demonstrates superiority over prompt-based methods, even without parameter expansion in each learning session. Motivated by this, we propose incrementally tuning the shared adapter without imposing parameter update constraints, enhancing the learning capacity of the backbone. Additionally, we employ feature sampling from stored prototypes to retrain a unified classifier, further improving its performance. We estimate the semantic shift of old prototypes without access to past samples and update stored prototypes session by session. Our proposed method eliminates model expansion and avoids retaining any image samples. It surpasses previous pre-trained model-based CIL methods and demonstrates remarkable continual learning capabilities. Experimental results on five CIL benchmarks validate the effectiveness of our approach, achieving state-of-the-art (SOTA) performance.

Pipeline

Results

Requirements

Dependencies

1. torch 1.11.0
2. torchvision 0.12.0
3. timm 0.6.12

Datasets

We provide the processed datasets as follows:

• CIFAR100: will be automatically downloaded by the code.
• CUB200 ImageNet-R ImageNet-A VTAB: Reference Revisiting

These subsets are sampled from the original datasets. Please note that we do not have the right to distribute these datasets. If the distribution violates the license, I shall provide the filenames instead.

You need to modify the path of the datasets in ./data/data.py according to your own path.

Training Scripts

Please follow the settings in the exps folder to prepare your json files, and then run:

python main.py --config ./exps/[configname].json

for imageneta:
python main.py --config ./exps/adapter_imageneta.json
for imagenetr:
python main.py --config ./exps/adapter_imagenetr.json
for cifar224:
python main.py --config ./exps/adapter_cifar224.json
for cub200:
python main.py --config ./exps/adapter_cub.json

Citation

If you find this useful in your research, please consider citing:

@article{tan2024semantically,
 title={Semantically-Shifted Incremental Adapter-Tuning is A Continual ViTransformer},
 author={Tan, Yuwen and Zhou, Qinhao and Xiang, Xiang and Wang, Ke and Wu, Yuchuan and Li, Yongbin},
 journal={arXiv preprint arXiv:2403.19979},
 year={2024}
}

Acknowledgment

This repo is based on RevisitingCIL and PyCIL.

The implemenations of parameter-efficient tuning methods are based on VPT, AdaptFormer, and SSF.

Thanks for their wonderful work!!!

Correspondence

If you have any question about this project, please contact xex@hust.edu.cn