Awesome

Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning

Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning<br> Jinglin Liang¹, Jin Zhong¹, Hanlin Gu², Zhongqi Lu³, Xingxing Tang², Gang Dai¹, Shuangping Huang^1,5*, Lixin Fan⁴, Qiang Yang^2,4 <br> ¹South China University of Technology, ²The Hong Kong University of Science and Technology, ³China University of Petroleum, ⁴WeBank, ⁵Pazhou Laboratory

Abstract: <br> Federated Class Continual Learning (FCCL) merges the challenges of distributed client learning with the need for seamless adaptation to new classes without forgetting old ones. The key challenge in FCCL is catastrophic forgetting, an issue that has been explored to some extent in Continual Learning (CL). However, due to privacy preservation requirements, some conventional methods, such as experience replay, are not directly applicable to FCCL. Existing FCCL methods mitigate forgetting by generating historical data through federated training of GANs or data-free knowledge distillation. However, these approaches often suffer from unstable training of generators or low-quality generated data, limiting their guidance for the model. To address this challenge, we propose a novel method of data replay based on diffusion models. Instead of training a diffusion model, we employ a pre-trained conditional diffusion model to reverse-engineer each category, searching the corresponding input conditions for each category within the model's input space, significantly reducing computational resources and time consumption while ensuring effective generation. Furthermore, we enhance the classifier's domain generalization ability on generated and real data through contrastive learning, indirectly improving the representational capability of generated data for real data. Extensive experiments demonstrate that our method significantly outperforms existing baselines.

📢 Description

This repository is the official PyTorch implementation of:

Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning (ECCV 2024 Oral).

🔨 Requirement

Environment

pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html

Then install CLIP from the official CLIP repository.

Prepare Data

The program will automatically download the CIFAR-100 dataset. You only need to download the Tiny ImageNet dataset using the following commands.

cd data
wget http://cs231n.stanford.edu/tiny-imagenet-200.zip
unzip tiny-imagenet-200.zip
python preprocess.py
cd ..

🍔 Pre-trained Model

We use the pretrained diffusion model from LDM repository, you can simply use the following command to obtain the pre-trained model.

mkdir -p models/ldm/text2img-large
wget -O models/ldm/text2img-large/model.ckpt https://ommer-lab.com/files/latent-diffusion/nitro/txt2img-f8-large/model.ckpt

Please download bert-base-uncased from here, and put it in models/bert.

🚀 Reproducing

Just set the GPU_LIST in rep_run.sh and run this command to easily reproduce most of the results from our paper.

bash rep_run.sh

You can also reproduce our method or baseline methods separately using the following commands.

# DDDR
python main.py --dataset cifar100 --method ours --tasks 5 --beta 0.5 --seed 2024

# Finetune
python main.py --dataset cifar100 --method finetune --tasks 10 --beta 0.5 --seed 2024

# EWC
python main.py --dataset cifar100 --method ewc --tasks 10 --beta 0.5 --seed 2024

# Target
python main.py --dataset cifar100 --method target --tasks 5  --beta 0.5 --seed 2024 --w_kd 25

🏰 Other related projects

• Textual Inversion
• Target

We would like to express our heartfelt gratitude for their contribution to our project.

❤️ Citation

If you find our work inspiring or use our codebase in your research, please cite our work:

@inproceedings{liang2024dddr,
  title={Diffusion-Driven Data Replay: A Novel Approach to Combat Forgetting in Federated Class Continual Learning},
  author={Liang, Jinglin and Zhong, Jin and Gu, Hanlin and Lu, Zhongqi and Tang, Xingxing and Dai, Gang and Huang, Shuangping and Fan, Lixin and Yang, Qiang},
  booktitle={ECCV},
  year={2024}
}