Home

Awesome

Demo project for training Swin with Mesa

For more details, please checkout our paper: Mesa: A Memory-saving Training Framework for Transformers.

Usage

  1. Install Mesa from here.
  2. Install dependencies of Swin from here.
  3. To train a Swin model.
    conda activate mesa
    
    # Make sure you have the correct path to the dataset in the config file.
    python -m torch.distributed.launch --nproc_per_node [GPUs] --master_port 13235  main.py \
    --cfg [configs]
    
    # For example, to train Swin-Ti with Mesa on 1 GPU.
    python -m torch.distributed.launch --nproc_per_node 1 --master_port 13235  main.py \
    --cfg configs/swin_tiny_patch4_window7_224_mesa.yaml
    

Results on ImageNet

ModelParam (M)FLOPs (G)Train MemoryTop-1 (%)
Swin-Ti294.511,81281.3
Swin-Ti w/ Mesa294.55,37181.3

Acknowledgments

This repository has adopted codes from Swin Transformer, we thank the authors for their open-sourced code.