Awesome

Which Transformer to Favor: <br>A Comparative Analysis of Efficiency in Vision Transformers

This is the code for the paper Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers, a benchmark of over 45+ different efficient vision trainsformers. We train models from scratch and track multiple efficiency metrics.

Abstract

Self-attention in Transformers comes with a high computational cost because of their quadratic computational complexity, but their effectiveness in addressing problems in language and vision has sparked extensive research aimed at enhancing their efficiency. However, diverse experimental conditions, spanning multiple input domains, prevent a fair comparison based solely on reported results, posing challenges for model selection. To address this gap in comparability, we perform a large-scale benchmark of more than 45 models for image classification, evaluating key efficiency aspects, including accuracy, speed, and memory usage. Our benchmark provides a standardized baseline for efficiency-oriented transformers. We analyze the results based on the Pareto front -- the boundary of optimal models. Surprisingly, despite claims of other models being more efficient, ViT remains Pareto optimal across multiple metrics. We observe that hybrid attention-CNN models exhibit remarkable inference memory- and parameter-efficiency. Moreover, our benchmark shows that using a larger model in general is more efficient than using higher resolution images. Thanks to our holistic evaluation, we provide a centralized resource for practitioners and researchers, facilitating informed decisions when selecting or developing efficient transformers.

Updates

• [28.09.2024] We have created a blogpost on the WTF Benchmark with interactive plots :computer_mouse: :computer:
• [10.09.2024] We are happy to announce that the benchmark has been accepted for publication at WACV 2025 :tada:
• [22.07.2024] We have released a new version of the paper on ArXiv featuring 12 new models and more analysis into the scaling behaviors :newspaper_roll:
• [19.07.2024] We have added AFNO, EfficientMod, EfficientViT, FastViT, HiViT, Hydra Attention, Informer, Next-ViT, Reformer, ResT, SLAB, and SwiftFormer to the benchmark :gear:
• [19.07.2024] We have uniformly reformatted most files :black_flag:

Requirements

This project heavily builds on timm and open source implementations of the models that are tested. All requirements are listed in requirements.txt. To install those, run

pip install -r requirements.txt

Usage

After cloning this repository, you can train and test a lot of different models. By default, a srun command is executed to run the code on a slurm cluster. To run on the local machine, append the -local flag to the command.

Dataset Preparation

Supported datasets are CIFAR10, ImageNet-21k, and ImageNet-1k.

The CIFAR10 dataset has to be located in a subfolder of the dataset root directory called CIFAR. This is the normal CIFAR10 from torchvision.datasets.

To speed up the data loading, the ImageNet datasets are read using datadings. The .msgpack files for ImageNet-1k should be located in <dataset_root_folder>/imagenet/msgpack, while the ones for ImageNet-21k should be in <dataset_root_folder>/imagenet-21k. See the datadings documentation for information on how to create those files.

Training

Pretraining

To pretrain a model on a given dataset, run

./main.py -model <model_name> -epochs <epochs> -dataset_root <dataset_root_folder>/ -results_folder <folder_for_results>/ -logging_folder <logging_folder> -run_name <name_or_description_of_the_run> (-local)

This will save a checkpoint (.tar file) every <save_epochs> epochs (the default is 10), which contains all the model weights, along with the optimizer and scheduler state, and the current training stats. The default pretraining dataset is ImageNet-21k.

Finetuning

A model (checkpoint) can be finetuned on another dataset using the following command:

./main.py -task fine-tune -model <model_checkpoint_file.tar> -epochs <epochs> -lr <lr> -dataset_root <dataset_root_folder>/ -results_folder <folder_for_results>/ -logging_folder <logging_folder> -run_name <name_or_description_of_the_run> (-local)

This will also save new checkpoints during training. The default finetuning dataset is ImageNet-1k.

Evaluation

It is also possible to evaluate the models. To evaluate the model's accuracy and the efficiency metrics, run

./main.py -task eval -model <model_checkpoint_file.tar> -dataset_root <dataset_root_folder>/ -results_folder <folder_for_results>/ -logging_folder <logging_folder> -run_name <name_or_description_of_the_run> (-local)

The default evaluation dataset is ImageNet-1k.

To only evaluate the efficiency metrics, run

./main.py -task eval-metrics -model <model_checkpoint_file.tar> -dataset_root <dataset_root_folder>/ -results_folder <folder_for_results>/ -logging_folder <logging_folder> -run_name <name_or_description_of_the_run> (-local)

This utilizes the CIFAR10 dataset by default.

Further Arguments

There can be multiple further arguments and flags given to the scripts. The most important ones are

For a list of all arguments, run

./main.py --help

Supported Models

These are the models we support. Links are to original code sources. If no link is provided, we implemented the architecture from scratch, following the specific paper.

License

We release this code under the MIT license.

Citation

If you use this codebase in your project, please cite:

@misc{Nauen2023WTFBenchmark,
      title={Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers}, 
      author={Tobias Christian Nauen and Sebastian Palacio and Andreas Dengel},
      year={2023},
      eprint={2308.09372},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      note={Accepted at WACV 2025}
}