Awesome
Overview
This is the official repository of A Theoretical Study on Solving Continual Learning (NeurIPS 2022).
In this repository, we provide the codes for Sup+CSI and Sup+CSI+c. For HAT+CSI and HAT+CSI+c, please check the repository of our workshop paper (Link)
Note
Check out our related paper
- A Multi-Head Model for Continual Learning via Out-of-Distribution Replay, CoLLAs 2022 [PDF]
Environments
The code has been tested on the machine with
RTX 3090
- cuda=11.4
- pytorch=1.7.1
- torchvision=0.8.2
- cudatoolkit=11.0.221
- tensorboardx=2.1
- diffdist=0.1
- gdown=4.4.0
Please install the necessary packages
Training
Please change --dataset --config --log_dir --name for different experiments
python experiments/GG/run.py --data='./data' --dataset 'cifar10' --config 'cifar10_5t_csi' --log_dir './logs/cifar10_5t_csi' --name 'cifar10_5t_csi' --ood_method 'csi' --round 0 --gpu-sets=0 --seeds 1 --save --original --amp
choices of dataset: mnist, cifar10, cifar100, timgnet choices of config: mnist_5t_csi, cifar10_5t_csi, cifar100_10t_csi, cifar100_20t_csi, timgnet_5t_csi, timgnet_10t_csi
Without mixed precision, remove --amp
To resume training,
python experiments/GG/run.py --data './data' --dataset 'cifar10' --config 'cifar10_5t_csi' --log_dir './logs/cifar10_5t_csi' --load_path './logs/cifar10_5t_csi/cifar10_5t_csi_round_0/result_joint_3.pt' --name 'cifar10_5t_csi' --ood_method 'csi' --round 0 --gpu-sets=0 --seeds 1 --resume_task 4 --original
Please change load_path and resume_task for your experiment.
Training calibration parameters
Please change --dataset, --config --log_dir, --load_path, --name, --calibration_task
python experiments/GG/run.py --data './data' --dataset 'cifar10' --config 'cifar10_5t_csi' --log_dir './logs/cifar10_5t_csi' --load_path './logs/cifar10_5t_csi/cifar10_5t_csi_round_0/result_joint_4.pt' --name 'cifar10_5t_csi' --print_filename 'calibration_4.txt' --ood_method 'csi' --round 0 --gpu-sets=0 --seeds 1 --original --calibration_task 4
The numbers for --calibration_task and result_joint_NUMBER.pt in --load_path must be the same
Evaluation without calibration
python experiments/GG/run.py --data './data' --dataset 'cifar10' --config 'cifar10_5t_csi' --log_dir './logs/cifar10_5t_csi' --load_path './logs/cifar10_5t_csi/cifar10_5t_csi_round_0/result_joint_4.pt' --name 'cifar10_5t_csi' --print_filename 'eval_without_cal.txt' --round 0 --gpu-sets=0 --seeds 1 --ood_method 'csi' --original
Evaluation using calibration
python experiments/GG/run.py --data './data' --dataset 'cifar10' --config 'cifar10_5t_csi' --log_dir './logs/cifar10_5t_csi' --load_path './logs/cifar10_5t_csi/cifar10_5t_csi_round_0/result_joint_4.pt' --name 'cifar10_5t_csi' --cal_pretrain './logs/cifar10_5t_csi/cifar10_5t_csi_round_0/calibration_4' --print_filename 'eval_with_cal.txt' --round 0 --gpu-sets=0 --seeds 1 --ood_method 'csi' --original
Evaluation using pre-trained models
Please download the pre-trained models and calibration parameters by running download_pretrained_models.py or download manually from link. The models and calibration parameters need to be saved under LOG_DIR/NAME_round_NO/, where LOG_DIR is --log_dir (e.g., ./logs/cifar10_5t_csi) and NAME is --name (e.g., cifar10_5t_csi) and NO is the round number (e.g., usually 0).
The provided pre-trained models give the following CIL results
| MNIST | CIFAR10 | CIFAR100-10T | CIFAR100-20t | T-ImageNet-5T | T-ImageNet-10T | |
|---|---|---|---|---|---|---|
| Sup+CSI | 79.62 | 85.52 | 64.63 | 59.28 | 49.25 | 45.81 |
| Sup+CSI+c | 82.51 | 87.75 | 64.67 | 59.42 | 49.52 | 45.96 |
Different prediction method (Appendix C)
For the CIL prediction method explained in Appendix C, please run
python cil_is_til_times_tp.py