Home

Awesome

Tuning Pre-trained Model via Moment Probing

This repo is the official implementation of ICCV 2023 paper "Tuning Pre-trained Model via Moment Probing", created by Mingze Gao, Qilong Wang and Zhenyi Lin.

Introduction

In this paper, we propose a Moment Probing (MP) method, which performs a linear classifier on powerful representations characterized by feature distribution to finetune pretrained vision backbones efficiently. A partially shared module is proposed to learn two recalibrating parameters (PSRP), resulting in a complete finetuning framework MP+.

<div align="center"> <img src="./figures/whole.png" width="57%"> <img src="./figures/Train_cost.png" width="42%"> </div>

Usage

Data preparation

wget https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz

Prepare ImageNet-A, ImageNet-R and ImageNet-C for evaluation.

Pre-trained model preparation

Tuning Pre-trained Model via Moment Probing

To fine-tune a pre-trained ViT model via MP or MP+ on ImageNet-1K, run:

bash train_scripts/vit/imagenet_1k/train_mp.sh
bash train_scripts/vit/imagenet_1k/train_mp+.sh

Following the scripts we provided, you can easily reproduce our results.

Robustness & OOD

To evaluate the performance of fine-tuned model via MP or MP+ on Robustness & OOD, run:

bash train_scripts/vit/imagenet_a(r, c)/eval_mp.sh

checkpoint list (ViT-B/16 SUP)

MethodAcc@1(%)#Params.(M)logCheckpoint
Linear Probing (LP)82.040.77logparams
Full Finetuning (FT)83.5886.57logparams
Moment Probing (MP)83.153.65logparams
Moment Probing+ (MP+)83.624.10logparams

TODO

Citation

@inproceedings{gao2023iccv,
  title={Tuning Pre-trained Model via Moment Probing},
  author={Mingze Gao and Qilong Wang and Zhenyi Lin and Pengfei Zhu and Qinghua Hu and Jingbo Zhou},
  booktitle = {ICCV},
  year={2023}
}

Acknowledgement

Our code are built following DeiT, Swin Transformer, ConvNeXt, CLIP, SSF , thanks for their excellent work