Awesome

Current applications

Classification: Please see <a href="https://github.com/ViTAE-Transformer/ViTAE-VSA/tree/main/Image-Classification">ViTAE-VSA for Image Classification</a> for usage detail;

Object Detection: Please see <a href="https://github.com/ViTAE-Transformer/ViTAE-VSA/tree/main/Object-Detection">ViTAE-VSA for Object Detection</a> for usage detail;

Semantic Segmentation: Will be released in next few days;

Other ViTAE applications

ViTAE & ViTAEv2: Please see <a href="https://github.com/ViTAE-Transformer/ViTAE-Transformer">ViTAE-Transformer for Image Classification, Object Detection, and Sementic Segmentation</a>;

Matting: Please see <a href="https://github.com/ViTAE-Transformer/ViTAE-Transformer-Matting">ViTAE-Transformer for matting</a>;

Remote Sensing: Please see <a href="https://github.com/ViTAE-Transformer/ViTAE-Transformer-Remote-Sensing">ViTAE-Transformer for Remote Sensing</a>;

Updates

19/09/2022

• The code and training logs for ViTAE-VSA have been released! The semantic segmentation and Swin+VSA will be relased in next few days.

09/07/2022

• The paper is accepted by ECCV'22!

19/04/2022

• The paper is post on arxiv! The code will be made public available once cleaned up.

Introduction

Usage

If you are interested in using the VSA attention only, please consider this file in classification or the VSAWindowAttention Class in object detection applications.

Classification Results

ViTAEv2* denotes the version using window attention for all stages, which have much less memory requirements anc computations.

Main Results on ImageNet-1K with pretrained models

Models with ImageNet-22K pretraining

Object Detection Results

ViTAEv2* denotes the version using window attention for all stages, which have much less memory requirements anc computations.

Mask R-CNN

Cascade Mask R-CNN

Semantic Segmentation Results for Cityscapes

ViTAEv2* denotes the version using window attention for all stages.

UperNet

512x1024 resolution for training and testing

769x769 resolution for training and testing

Please refer to our paper for more experimental results.

Statement

This project is for research purpose only. For any other questions please contact qmzhangzz at hotmail.com yufei.xu at outlook.com.

The code base is borrowed from T2T, ViTAEv2 and Swin.

Citing VSA and ViTAE

@article{zhang2022vsa,
  title={VSA: Learning Varied-Size Window Attention in Vision Transformers},
  author={Zhang, Qiming and Xu, Yufei and Zhang, Jing and Tao, Dacheng},
  journal={arXiv preprint arXiv:2204.08446},
  year={2022}
}
@article{zhang2022vitaev2,
  title={ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond},
  author={Zhang, Qiming and Xu, Yufei and Zhang, Jing and Tao, Dacheng},
  journal={arXiv preprint arXiv:2202.10108},
  year={2022}
}
@article{xu2021vitae,
  title={Vitae: Vision transformer advanced by exploring intrinsic inductive bias},
  author={Xu, Yufei and Zhang, Qiming and Zhang, Jing and Tao, Dacheng},
  journal={Advances in Neural Information Processing Systems},
  volume={34},
  year={2021}
}