Home

Awesome

CrossFormer++

This repository is the code for our papers:

The crossformer branch retains the old version code with old dependencies.

Updates

Introduction

Existing vision transformers fail to build attention among objects/features of different scales (cross-scale attention), while such ability is very important to visual tasks. CrossFormer is a versatile vision transformer which solves this problem. Its core designs contain Cross-scale Embedding Layer (CEL), Long-Short Distance Attention (L/SDA), which work together to enable cross-scale attention.

CEL blends every input embedding with multiple-scale features. L/SDA split all embeddings into several groups, and the self-attention is only computed within each group (embeddings with the same color border belong to the same group.).

Besides, we also propose a Dynamic Position Bias (DPB) module, which makes the effective yet inflexible relative position bias apply to variable image size.

Further, in CrossFormer++, we introduce a Progressive Group Size (PGS) strategy to achieve a better balance between performance and computation budget and a Activation Cooling Layer (ACL) to suppress the magnitude of activations that grows drastically in the residual stream.

Now, experiments are done on four representative visual tasks, i.e., image classification, objection detection, and instance/semantic segmentation. Results show that CrossFormer outperforms existing vision transformers in these tasks, especially in dense prediction tasks (i.e., object detection and instance/semantic segmentation). We think it is because image classification only pays attention to one object and large-scale features, while dense prediction tasks rely more on cross-scale attention.

Prerequisites

  1. Create and activate conda environment
conda create -n crossformer_env python=3.9 -y
conda activate crossformer_env
  1. Libraries (Python3.9-based)
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch
pip install tensorboard termcolor
pip install timm pyyaml yacs protobuf==3.20.0
  1. Dataset: ImageNet

  2. Requirements for detection/instance segmentation and semantic segmentation are listed here: detection/README.md or segmentation/README.md

For ease of use, we have adapted our code of CrossFormer and CrossFormer++ with newer version of pytorch, mmcv, mmdetection and mmsegmentation, so the results in this repository may be slightly different from the results reported in the paper.

If you're using relatively old versions of CUDA, please consider using our original CrossFormer code at crossformer branch of this repository.

Getting Started

Training

## There should be two directories under the path_to_imagenet: train and validation

## CrossFormer-T
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer/tiny_patch4_group7_224.yaml \
--batch-size 128 --data-path path_to_imagenet --output ./output

## CrossFormer-S
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer/small_patch4_group7_224.yaml \
--batch-size 128 --data-path path_to_imagenet --output ./output

## CrossFormer-B
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer/base_patch4_group7_224.yaml 
--batch-size 128 --data-path path_to_imagenet --output ./output

## CrossFormer-L
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer/large_patch4_group7_224.yaml \
--batch-size 128 --data-path path_to_imagenet --output ./output

## CrossFormer++-S
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer_pp/small_patch4_group_const_224.yaml \
--batch-size 128 --data-path path_to_imagenet --output ./output

## CrossFormer++-B
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer_pp/base_patch4_group_const_224.yaml \
--batch-size 128 --data-path path_to_imagenet --output ./output

## CrossFormer++-L
python -u -m torch.distributed.launch --nproc_per_node 8 main.py --cfg configs/crossformer_pp/large_patch4_group_const_224.yaml \
--batch-size 128 --data-path path_to_imagenet --output ./output

Testing

## Take CrossFormer-T as an example of evaluating accuracy
python -u -m torch.distributed.launch --nproc_per_node 1 main.py --cfg configs/crossformer/small_patch4_group7_224.yaml \
--batch-size 128 --data-path path_to_imagenet --eval --resume path_to_crossformer-t.pth

## Take CrossFormer-T as an example of testing throughput
python -u -m torch.distributed.launch --nproc_per_node 1 main.py --cfg configs/crossformer/small_patch4_group7_224.yaml \
--batch-size 128 --data-path path_to_imagenet --throughput

You need to modify the path_to_imagenet and path_to_crossformer-t.pth accordingly.

Training and testing scripts for objection detection: detection/README.md.

Training and testing scripts for semantic segmentation: segmentation/README.md.

Results

Image Classification

Models trained on ImageNet-1K and evaluated on its validation set. The input image size is 224 x 224.

ArchitecturesParamsFLOPsAccuracyModels
ResNet-5025.6M4.1G76.2%-
RegNetY-8G39.0M8.0G81.7%-
CrossFormer-T27.8M2.9G81.5%Google Drive/BaiduCloud, key: nkju
CrossFormer-S30.7M4.9G82.5%Google Drive/BaiduCloud, key: fgqj
CrossFormer++-S23.3M4.9G83.2%BaiduCloud, key:crsf
CrossFormer-B52.0M9.2G83.4%Google Drive/BaiduCloud, key: 7md9
CrossFormer++-B52.0M9.5G84.2%BaiduCloud, key:crsf
CrossFormer-L92.0M16.1G84.0%Google Drive/BaiduCloud, key: cc89
CrossFormer++-L92.0M16.6G84.7%BaiduCloud, key:crsf
CrossFormer++-H96.0M21.8G84.9%BaiduCloud, key:crsf

More results compared with other vision transformers can be seen in the paper.

Note: Checkpoints of CrossFormer++ will be released as soon as possible.

Objection Detection & Instance Segmentation

Models trained on COCO 2017. Backbones are initialized with weights pre-trained on ImageNet-1K.

BackboneDetection HeadLearning ScheduleParamsFLOPsbox APmask AP
ResNet-101RetinaNet1x56.7M315.0G38.5-
CrossFormer-SRetinaNet1x40.8M282.0G44.4-
CrossFormer++-SRetinaNet1x40.8M282.0G45.1-
CrossFormer-BRetinaNet1x62.1M389.0G46.2-
CrossFormer++-BRetinaNet1x62.2M389.0G46.6-
BackboneDetection HeadLearning ScheduleParamsFLOPsbox APmask AP
ResNet-101Mask-RCNN1x63.2M336.0G40.436.4
CrossFormer-SMask-RCNN1x50.2M301.0G45.441.4
CrossFormer++-SMask-RCNN1x43.0M287.4G46.442.1
CrossFormer-BMask-RCNN1x71.5M407.9G47.242.7
CrossFormer++-BMask-RCNN1x71.5M408.0G47.743.2
<!-- | **CrossFormer-S** | Mask-RCNN | 3x | **50.2M** | **291.1G** | **48.7** | **43.9** | | **CrossFormer-B** | Mask-RCNN | 3x | **71.5M** | **398.1G** | **49.8** | **44.5** | | **CrossFormer-S** | Cascade-Mask-RCNN | 3x | **88.0M** | **769.7G** | **52.2** | **45.2** | -->

More results and pretrained models for objection detection: detection/README.md.

Semantic Segmentation

Models trained on ADE20K. Backbones are initialized with weights pre-trained on ImageNet-1K.

BackboneSegmentation HeadIterationsParamsFLOPsIOUMS IOU
CrossFormer-SFPN80K34.3M209.8G46.4-
CrossFormer++-SFPN80K27.1M199.5G47.4-
CrossFormer-BFPN80K55.6M320.1G48.0-
CrossFormer++-BFPN80K55.6M331.1G48.6-
CrossFormer-LFPN80K95.4M482.7G49.1-
BackboneSegmentation HeadIterationsParamsFLOPsIOUMS IOU
ResNet-101UPerNet160K86.0M1029.G44.9-
CrossFormer-SUPerNet160K62.3M979.5G47.648.4
CrossFormer++-SUPerNet160K53.1M963.5G49.450.8
CrossFormer-BUPerNet160K83.6M1089.7G49.750.6
CrossFormer++-BUPerNet160K83.7M1089.8G50.751.0
CrossFormer-LUPerNet160K125.5M1257.8G50.451.4

MS IOU means IOU with multi-scale testing.

More results and pretrained models for semantic segmentation: segmentation/README.md.

Citing Us

@inproceedings{wang2021crossformer,
  title = {CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention},
  author = {Wenxiao Wang and Lu Yao and Long Chen and Binbin Lin and Deng Cai and Xiaofei He and Wei Liu},
  booktitle = {International Conference on Learning Representations, {ICLR}},
  url = {https://openreview.net/forum?id=_PHymLIxuI},
  year = {2022}
}

@article{wang2023crossformer++,
  author       = {Wenxiao Wang and Wei Chen and Qibo Qiu and Long Chen and Boxi Wu and Binbin Lin and Xiaofei He and Wei Liu},
  title        = {Crossformer++: A versatile vision transformer hinging on cross-scale attention},
  journal      = {{IEEE} Transactions on Pattern Analysis and Machine Intelligence, {TPAMI}},
  year         = {2023},
  doi          = {10.1109/TPAMI.2023.3341806},
}

Acknowledgement

Part of the code of this repository refers to Swin Transformer.