Awesome
PyTorch Implementation of Stand-Alone Self-Attention in Vision Models
This is NOT an official implementation. Please let me know whether this implementation contains any misreadings of the original paper.
Prerequisites
- Python +3.6
- pytorch +1.1.0
- scipy
- Pillow
- torchvision
Benchmark (WIP)
Trained with ImageNet. (WIP: CIFAR-10, CIFAR-100)
Backbone network and parameters are based on the official torchvision ResNet and trainer example.
Trained up to 90 epochs / batch 64 on a single NVIDIA 1080Ti GPU, with SGD optimizer with a learning rate of 0.1 which is linearly warmed up for 10 epochs followed by cosine decay. (according to the SASA paper)