Home

Awesome

Enhancing the Performance of Transformer-based Spiking Neural Networks by SNN-optimized Downsampling with Precise Gradient Backpropagation, Arxiv 2023

SGLFormer: Spiking Global-Local-Fusion Transformer with high performance, This link

Our CML models achieve the state-of-the-art performance on several datasets (eg. 77.64 % on ImageNet, 96.04 % on CIFAR10, 80.37 % on CIFAR100, 81.4% on CIFAR10-DVS) in directly trained spiking neural networks in 2023/05. Our model achieves 78.46 % on ImageNet with 288 * 288 resolution.

Our newly improved model with CML, named SGLFormer, has achieved SOTA performance on several datasets (eg. 83.73 % on ImageNet, 96.76 % on CIFAR10, 82.26 % on CIFAR100, 82.9% on CIFAR10-DVS) in directly trained SNNs in 2024/03.

<br> <br> <p align="center"> <img src="https://github.com/zhouchenlin2096/Spikingformer-CML/blob/master/imgs/SNN-optimized-downsampling.png"> </p>

News

[2024.3.12] Our work with CML has been accepted in Frontiers in Neuroscience 2024.

[2023.9.11] Update origin_logs.

[2023.8.18] Update trained models.

Reference

If you find this repo useful, please consider citing:

@ARTICLE{10.3389/fnins.2024.1371290,
  AUTHOR={Zhang, Han  and Zhou, Chenlin  and Yu, Liutao  and Huang, Liwei  and Ma, Zhengyu  and Fan, Xiaopeng  and Zhou, Huihui  and Tian, Yonghong },
  TITLE={SGLFormer: Spiking Global-Local-Fusion Transformer with high performance},
  JOURNAL={Frontiers in Neuroscience},
  VOLUME={18},
  YEAR={2024},
  URL={https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2024.1371290},
  DOI={10.3389/fnins.2024.1371290},
  ISSN={1662-453X}
}

@misc{zhou2023enhancing,
      title={Enhancing the Performance of Transformer-based Spiking Neural Networks by Improved Downsampling with Precise Gradient Backpropagation}, 
      author={Chenlin Zhou and Han Zhang and Zhaokun Zhou and Liutao Yu and Zhengyu Ma and Huihui Zhou and Xiaopeng Fan and Yonghong Tian},
      year={2023},
      eprint={2305.05954},
      archivePrefix={arXiv},
      primaryClass={cs.NE}
}

Main results on ImageNet-1K

ModelResolutionTParam.Top-1 AccDownload
CML + Spikformer-8-384224x224416.81M72.73-
CML + Spikformer-8-512224x224429.68M75.61-
CML + Spikformer-8-768224x224466.34M77.34-
CML + Spikingformer-8-384224x224416.81M74.35-
CML + Spikingformer-8-512224x224429.68M76.54-
CML + Spikingformer-8-768224x224466.34M77.64here
CML + Spikingformer-8-768288x288466.34M78.46-

Download password: abcd

Main results on CIFAR10/CIFAR100

ModelTParam.CIFAR10 Top-1 AccDownloadCIFAR100 Top-1 Acc
CML + Spikformer-4-25644.15M94.82-77.64
CML + Spikformer-2-38445.76M95.63-78.75
CML + Spikformer-4-38449.32M95.93-79.65
CML + Spikformer-4-384-400E49.32M96.04-80.02
CML + Spikingformer-4-25644.15M94.94-78.19
CML + Spikingformer-2-38445.76M95.54-78.87
CML + Spikingformer-4-38449.32M95.81-79.98
CML + Spikingformer-4-384-400E49.32M95.95here80.37

Download password: abcd

Main results on CIFAR10-DVS/DVS128

ModelTParam.CIFAR10 DVS Top-1 AccDVS 128 Top-1 Acc
CML + Spikformer-2-256102.57M79.297.6
CML + Spikformer-2-256162.57M80.998.6
CML + Spikingformer-2-256102.57M80.597.2
CML + Spikingformer-2-256162.57M81.498.6

Requirements

timm==0.6.12; cupy==11.4.0; torch==1.12.1; spikingjelly==0.0.0.0.12; pyyaml;

data prepare: ImageNet with the following folder structure, you can extract imagenet by this script.

│imagenet/
├──train/
│  ├── n01440764
│  │   ├── n01440764_10026.JPEG
│  │   ├── n01440764_10027.JPEG
│  │   ├── ......
│  ├── ......
├──val/
│  ├── n01440764
│  │   ├── ILSVRC2012_val_00000293.JPEG
│  │   ├── ILSVRC2012_val_00002138.JPEG
│  │   ├── ......
│  ├── ......

Train

Training on ImageNet

Setting hyper-parameters in imagenet.yml

cd imagenet
python -m torch.distributed.launch --nproc_per_node=8 train.py

Testing ImageNet Val data

Download the trained model first here.

cd imagenet
python test.py

Training on CIFAR10

Setting hyper-parameters in cifar10.yml

cd cifar10
python train.py

Training on CIFAR100

Setting hyper-parameters in cifar100.yml

cd cifar10
python train.py

Training on DVS128 Gesture

cd dvs128-gesture
python train.py

Training on CIFAR10-DVS

cd cifar10-dvs
python train.py

Acknowledgement & Contact Information

Related project: Spikingformer, spikformer, pytorch-image-models, spikingjelly.

For help or issues using this git, please submit a GitHub issue.

For other communications related to this git, please contact zhouchl@pcl.ac.cn or zhouchenlin19@mails.ucas.ac.cn.