Awesome

This codebase has developed into a new project that is well-maintained and includes more SOTA methods. Please refer to PyCIL: A Python Toolbox for Class-Incremental Learning for more information.

Implementation of continual learning methods

This repository implements some continual / incremental / lifelong learning methods by PyTorch.

Especially the methods based on memory replay.

• iCaRL: Incremental Classifier and Representation Learning. [paper]
• End2End: End-to-End Incremental Learning. [paper]
• DR: Lifelong Learning via Progressive Distillation and Retrospection. [paper]
• UCIR: Learning a Unified Classifier Incrementally via Rebalancing. [paper]
• BiC: Large Scale Incremental Learning. [paper]
• LwM: Learning without Memorizing. [paper]
• PODNet: PODNet: Pooled Outputs Distillation for Small-Tasks Incremental Learning. [paper]

Dependencies

1. torch 1.7.1
2. torchvision 0.8.2
3. tqdm
4. numpy
5. scipy

Usage

Run experiment

1. Edit the config.json file for global settings.
2. Edit the hyperparameters in the corresponding .py file (e.g., models/icarl.py).
3. Run:

python main.py

Add datasets

1. Add corresponding classes to utils/data.py.
2. Modify the _get_idata function in utils/data_manager.py.

Results

iCaRL

CIFAR100

<img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/iCaRL_cifar100_10.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/iCaRL_cifar100_20.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/iCaRL_cifar100_50.png" width = "325"/>

Average accuracies of CIFAR-100 (iCaRL):

UCIR

CIFAR100

<img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_CNN_cifar100_5.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_NCM_cifar100_5.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_CNN_cifar100_10.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_NCM_cifar100_10.png" width = "325"/>

ImageNet-Subset

<img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_CNN_imagenet_subset_5.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_NME_imagenet_subset_5.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_CNN_imagenet_subset_10.png" width = "325"/><img src="https://github.com/zhchuu/continual-learning-reproduce/blob/master/resources/UCIR_NME_imagenet_subset_10.png" width = "325"/>

BiC

ImageNet-1000

PODNet

CIFAR100

NME results are shown and the reproduced results are not in line with the reported results. Maybe I missed something...

Change log

• (2020.6.8) Store the data with list instead of np.array to avoid bugs when the image size is different.
• (2020.7.15) Avoid duplicative selection in constructing exemplars.
• (2020.10.3) Fix the bug of excessive memory usage.
• (2020.10.8) Store the data with np.array instead of Python list to obtain faster I/O.

Some problems

Q: Why can't I reproduce the results of the paper by this repository?

A: The result of the methods may be affected by the incremental order (In my opinion). You can either generate more orders and average their results or increase the number of training iterations (Adjust the hyperparameters).

References

https://github.com/arthurdouillard/incremental_learning.pytorch