Home

Awesome

CTP

GitHub Python PyTorch

The repository is

  1. A PyTorch library that provides some continual learning baseline algorithms on the vision-language continual pretraining benchmark P9D dataset.

  2. PyTorch implement for ICCV23 paper of "CTP: Towards Vision-Language Continual Pretraining via Compatible Momentum Contrast and Topology Preservation".

🎨 Introduction

Vision-Language Pretraining (VLP) has shown impressive results on diverse downstream tasks by offline training on large-scale datasets. Regarding the growing nature of real-world data, such an offline training paradigm on ever-expanding data is unsustainable, because models lack the continual learning ability to accumulate knowledge constantly. However, most continual learning studies are limited to uni-modal classification and existing multi-modal datasets cannot simulate continual non-stationary data stream scenarios.

To support the study of Vision-Language Continual Pretraining (VLCP), we first contribute a comprehensive and unified benchmark dataset P9D which contains over one million product image-text pairs from 9 industries. The data from each industry as an independent task supports continual learning and conforms to the real-world long-tail nature to simulate pretraining on web data.

We comprehensively study the characteristics and challenges of VLCP, and propose a new algorithm: Compatible momentum contrast with Topology Preservation, dubbed CTP. The compatible momentum model absorbs the knowledge of the current and previous-task models to flexibly update the modal feature. Moreover, Topology Preservation transfers the knowledge of embedding across tasks while preserving the flexibility of feature adjustment.

<p align="center"><img src="figs/main.png" alt="main" width="800"/></p>

⚙️ Setup and Environments

The detailed dependencies can refer to requirements.txt.

pip install -r requirements.txt

:clipboard: Reimplemented Methods

The detailed introduction of each baseline method can refer to the appendix of our paper or the corresponding raw paper.

Besides, the train folder provides the reimplemented codes on the vision-language continual pretraining task. Meanwhile, we also provide the training log of all methods as supplementary.

Memory-Free methods: Baseline methods without exemplar replay.

Memory-Buffer methods: Baseline methods with exemplar replay.

:file_folder: Dataset

The details of P9D dataset can be found in this repository.

📥 Pretrained Model Weight

The model weights of each method are too large. For example, each baseline method has model weights obtained from 8 tasks (2.3G*8=18.4G). As an alternative, we provide the training log of all baseline methods in the default and reversed task order.

Meanwhile, we provide the download links of CTP and CTP_ER model weights which are trained in the default task order. The link to Google Driver only has the model weights of the final task, but the link to Baidu Netdisk has the model weights of each task.

training logmodel weights
Google DriverHereHere
Baidu NetdiskHereHere

:mag: Train and Evaluation

Training from Scratch:

  1. Modify the file paths of the dataset in configs/base_seqF.yaml to your path.
  2. Modify the LOG_NAME and OUT_DIR in shell/seq_xxx.sh to your storage path. xxx represents the name of the method.
  3. Change the current path to the shell folder, and run the corresponding scripts seq_xxx.sh.
    cd /shell/
    sh seq_xxx.sh
    
  4. The corresponding training log will be written in the logger folder.

Evaluation:

  1. Modify the LOG_NAME and OUT_DIR in eval.sh to the storage path of the trained model.
  2. Run the evaluation script eval.sh.
    sh eval.sh
    

:pencil: Citation

If this codebase is useful to you, please consider citing:

@article{zhu2023ctp,
  title={CTP: Towards Vision-Language Continual Pretraining via Compatible
Momentum Contrast and Topology Preservation},
  author={Hongguang Zhu and Yunchao Wei and Xiaodan Liang and Chunjie Zhang and Yao Zhao},
  journal={Proceedings of the IEEE International Conference on Computer Vision},
  year={2023},
}

:panda_face: Contacts

If you have any questions, please feel free to contact me: zhuhongguang1103@gmail.com or hongguang@bjtu.edu.cn.

:books: Reference

  1. Li, Junnan, et al. "Align before Fuse: Vision and Language Representation Learning with Momentum Distillation." NeurIPS. 2021.
  2. Masana, Marc, et al. "Class-Incremental Learning: Survey and Performance Evaluation on Image Classification." TPAMI. 2023.
  3. Zhou, Dawei, et al. "PyCIL: a Python toolbox for class-incremental learning." SCIENCE CHINA Information Sciences. 2023.
  4. Hong, Xiaopeng, et al. "An Incremental Learning, Continual Learning, and Life-Long Learning Repository". Github repository
  5. Wang, Liyuan, et al. "A Comprehensive Survey of Continual Learning: Theory, Method and Application". arxiv 2023