Awesome
Energy-based Latent Aligner for Incremental Learning [arXiv | Poster | Video]
Accepted to CVPR 2022
<p align="center" width="100%"> <img src="https://user-images.githubusercontent.com/4231550/159659561-17bea6a6-5228-42e6-a811-eb18d37c48e9.png" width="500"/> </p> <p align="center" width="80%"> We illustrate an Incremental Learning model trained on a continuum of tasks in the top part of the figure. While learning the current task <img src="https://render.githubusercontent.com/render/math?math=\tau_t">, the latent representation of Task <img src="https://render.githubusercontent.com/render/math?math=\tau_{t-1}"> data gets disturbed, as shown by red arrows. ELI learns an energy manifold, and uses it to counteract this inherent representational shift, as illustrated by green arrows, thereby alleviating forgetting.</p>Overview
In this work, we propose ELI: Energy-based Latent Aligner for Incremental Learning, which:
- Learns an energy manifold for the latent representations such that previous task latents will have low energy and the current task latents have high energy values.
- This learned manifold is used to counter the representational shift that happens during incremental learning.
The implicit regularization that is offered by our proposed methodology can be used as a plug-and-play module in existing incremental learning methodologies for classification and object-detection.
Toy Experiment
<p align="center" width="100%"> <img src="https://user-images.githubusercontent.com/4231550/159659669-be756c6b-1948-4cd1-9ab7-acec9c69030b.png"/> </p>A key hypothesis that we base our methodology is that while learning a new task, the latent representations will get disturbed, which will in-turn cause catastrophic forgetting of the previous task, and that an energy manifold can be used to align these latents, such that it alleviates forgetting.
Here, we illustrate a proof-of-concept that our hypothesis is indeed true. We consider a two task experiment on MNIST, where each task contains a subset of classes: <img src="https://render.githubusercontent.com/render/math?math=\tau_1"> = {0, 1, 2, 3, 4}, <img src="https://render.githubusercontent.com/render/math?math=\tau_2"> = {5, 6, 7, 8, 9}.
After learning the second task, the accuracy on <img src="https://render.githubusercontent.com/render/math?math=\tau_1"> test set drops to 20.88%, while experimenting with a 32 dimensional latent space. The latent aligner in ELI provides 62.56% improvement in test accuracy to 83.44%. The visualization of a 512 dimensional latent space after learning <img src="https://render.githubusercontent.com/render/math?math=\tau_2"> in sub-figure (c), indeed shows cluttering due to representational shift. ELI is able to align the latents as shown in sub-figure (d), which alleviates the drop in accuracy from 89.14% to 99.04%.
The code for these toy experiments are in:
Implicitly Recognizing and Aligning Important Latents
https://user-images.githubusercontent.com/4231550/159675403-f2cee8e3-bddb-4e8f-80a1-90cb638b372e.mp4
Each row <img src="https://render.githubusercontent.com/render/math?math=i"> shows how <img src="https://render.githubusercontent.com/render/math?math=i^th"> latent dimension is updated by ELI. We see that different dimensions have different degrees of change, which is implicitly decided by our energy-based model.
Classification and Detection Experiments
Code and models for the classification and object detection experiments are inside the respective folders:
Each of these are independent repositories. Please consider them separate.
Citation
If you find our research useful, please consider citing us:
@inproceedings{joseph2022Energy,
title={Energy-based Latent Aligner for Incremental Learning},
author={Joseph, KJ and Khan, Salman and Khan, Fahad Shahbaz and Anwar, Rao Muhammad and Balasubramanian, Vineeth},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2022}
}