Home

Awesome

<div align="center"> <h1>MaPeT: Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training </h1>

arXiv

</div>

Lorenzo Baraldi*, Roberto Amoroso*, Marcella Cornia, Lorenzo Baraldi, Andrea Pilzer, Rita Cucchiara

* Equal contribution.

This is the official repository for the paper Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training.

Overview

<p align="center"> <img src="images/model.png" style="max-width:800px"> </p>

Abstract: <br> The use of self-supervised pre-training has emerged as a promising approach to enhance the performance of visual tasks such as image classification. In this context, recent approaches have employed the Masked Image Modeling paradigm, which pre-trains a backbone by reconstructing visual tokens associated with randomly masked image patches. This masking approach, however, introduces noise into the input data during pre-training, leading to discrepancies that can impair performance during the fine-tuning phase. Furthermore, input masking neglects the dependencies between corrupted patches, increasing the inconsistencies observed in downstream fine-tuning tasks. To overcome these issues, we propose a new self-supervised pre-training approach, named Masked and Permuted Vision Transformer (MaPeT), that employs autoregressive and permuted predictions to capture intra-patch dependencies. In addition, MaPeT employs auxiliary positional information to reduce the disparity between the pre-training and fine-tuning phases. In our experiments, we employ a fair setting to ensure reliable and meaningful comparisons and conduct investigations on multiple visual tokenizers, including our proposed k-CLIP which directly employs discretized CLIP features. Our results demonstrate that MaPeT achieves competitive performance on ImageNet, compared to baselines and competitors under the same model setting.

Citation

If you make use of our work, please cite our paper:

@article{baraldi2023learning,
      title={Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training}, 
      author={Lorenzo Baraldi and Roberto Amoroso and Marcella Cornia and Lorenzo Baraldi and Andrea Pilzer and Rita Cucchiara},
      journal={arXiv preprint arXiv:2306.07346},
      year={2023},
}

Getting Started

Follow these steps to get started with the project:

  1. Create a new conda environment: conda create -n mapet python=3.8.16
  2. Activate the environment: conda activate mapet
  3. Clone the repository: git clone https://github.com/aimagelab/MaPeT.git
  4. Change directory to the project root: cd MaPeT
  5. Install the required dependencies: pip install -r requirements.txt

To run validation with the default parameters, run the following command:

python -u validate.py <PathToImageNet> --model <model_name> --checkpoint <checkpoint_path> --interpolation "bicubic" --amp --gp "avg" --pin-mem --no-prefetcher

where:

External Code employed

This project makes use of code on the following external code repositories:

TODO

Acknowledgements

We thank CINECA for providing computational resources. This work has partially been supported by the projects PNRR-M4C2 (PE00000013) "FAIR - Future Artificial Intelligence Research" funded by the European Commission and "ELSA - European Lighthouse on Secure and Safe AI" funded by the EU (GA 101070617).