Awesome
MobileViTv3 : Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features [arXiv]
This repository contains MobileViTv3's source code for training and evaluation. It uses the CVNets library and is inspired by MobileViT (paper, code).
Installation and Training Models:
We recommend to use Python 3.8+ and PyTorch (version >= v1.8.0) with conda environment.
For setting-up the python environment with conda, see here.
MobileViTv3-S,XS,XXS
Download MobileViTv1 and replace the files provided in MobileViTv3-v1.
Conda environment used for training: environment_cvnet.yml.
Then install according to instructions provided in the downloaded repository.
For training, use training-and-evaluation readme provided in the downloaded repository.
MobileViTv3-1.0,0.75,0.5
Download MobileViTv2 and replace the files provided in MobileViTv3-v2.
Conda environment used for training: environment_mbvt2.yml
Then install according to instructions provided in the downloaded repository.
For training, use training-and-evaluation readme provided in the downloaded repository.
Trained models:
Download the trained MobileViTv3 models from here.
checkpoint_ema_best.pt files inside the model folder is used to generated the accuracy of models.
Low-latency models are build by reducing the number of MobileViTv3-blocks in 'layer4' from 4 to 2.
Please refer to the paper for more details.
Note that for the segmentation and detection, only the backbone architecture parameters are listed.
Classification
ImageNet-1K:
| Model name | Accuracy (%) | Parameters (Million) | FLOPs (Million) | Foldername |
|---|---|---|---|---|
| MobileViTv3-S | 79.3 | 5.8 | 1841 | mobilevitv3_S_e300_7930 |
| MobileViTv3-XS | 76.7 | 2.5 | 927 | mobilevitv3_XS_e300_7671 |
| MobileViTv3-XXS | 70.98 | 1.2 | 289 | mobilevitv3_XXS_e300_7098 |
| MobileViTv3-1.0 | 78.64 | 5.1 | 1876 | mobilevitv3_1_0_0 |
| MobileViTv3-0.75 | 76.55 | 3.0 | 1064 | mobilevitv3_0_7_5 |
| MobileViTv3-0.5 | 72.33 | 1.4 | 481 | mobilevitv3_0_5_0 |
ImageNet-1K using low-latency models:
| Model name | Accuracy (%) | Parameters (Million) | FLOPs (Million) | Foldername |
|---|---|---|---|---|
| MobileViTv3-S-L2 | 79.06 | 5.2 | 1651 | mobilevitv3_S_L2_e300_7906 |
| MobileViTv3-XS-L2 | 76.10 | 2.3 | 853 | mobilevitv3_XS_L2_e300_7610 |
| MobileViTv3-XXS-L2 | 70.23 | 1.1 | 256 | mobilevitv3_XXS_L2_e300_7023 |
Segmentation
PASCAL VOC 2012:
| Model name | mIoU | Parameters (Million) | Foldername |
|---|---|---|---|
| MobileViTv3-S | 79.59 | 7.2 | mobilevitv3_S_voc_e50_7959 |
| MobileViTv3-XS | 78.77 | 3.3 | mobilevitv3_XS_voc_e50_7877 |
| MobileViTv3-XXS | 74.04 | 2.0 | mobilevitv3_XXS_voc_e50_7404 |
| MobileViTv3-1.0 | 80.04 | 13.6 | mobilevitv3_voc_1_0_0 |
| MobileViTv3-0.5 | 76.48 | 6.3 | mobilevitv3_voc_0_5_0 |
ADE20K:
| Model name | mIoU | Parameters (Million) | Foldername |
|---|---|---|---|
| MobileViTv3-1.0 | 39.13 | 13.6 | mobilevitv3_ade20k_1_0_0 |
| MobileViTv3-0.75 | 36.43 | 9.7 | mobilevitv3_ade20k_0_7_5 |
| MobileViTv3-0.5 | 33.57 | 6.4 | mobilevitv3_ade20k_0_5_0 |
Detection MS-COCO:
| Model name | mAP | Parameters (Million) | Foldername |
|---|---|---|---|
| MobileViTv3-S | 27.3 | 5.5 | mobilevitv3_S_coco_e200_2730 |
| MobileViTv3-XS | 25.6 | 2.7 | mobilevitv3_XS_coco_e200_2560 |
| MobileViTv3-XXS | 19.3 | 1.5 | mobilevitv3_XXS_coco_e200_1930 |
| MobileViTv3-1.0 | 27.0 | 5.8 | mobilevitv3_coco_1_0_0 |
| MobileViTv3-0.75 | 25.0 | 3.7 | mobilevitv3_coco_0_7_5 |
| MobileViTv3-0.5 | 21.8 | 2.0 | mobilevitv3_coco_0_5_0 |
Citation
If you find this repository useful, please consider giving a star :star: and citation :mega::
@inproceedings{wadekar2022mobilevitv3,
title = {MobileViTv3: Mobile-Friendly Vision Transformer with Simple and Effective Fusion of Local, Global and Input Features},
author = {Wadekar, Shakti N. and Chaurasia, Abhishek},
doi = {10.48550/ARXIV.2209.15159},
year = {2022}
}