Awesome
LambdaNetworks: Modeling long-range Interactions without Attention
Experimnets (CIFAR10)
| Model | k | h | u | m | Params (M) | Acc (%) |
|---|
| ResNet18 baseline (ref) | | | | | 14 | 93.02 |
| LambdaResNet18 | 16 | 4 | 4 | 9 | 8.6 | 92.21 (70 Epochs) |
| LambdaResNet18 | 16 | 4 | 4 | 7 | 8.6 | 94.20 (67 Epochs) |
| LambdaResNet18 | 16 | 4 | 4 | 5 | 8.6 | 91.58 (70 Epochs) |
| LambdaResNet18 | 16 | 4 | 1 | 23 | 8 | 91.36 (69 Epochs) |
| ResNet50 baseline (ref) | | | | | 23.5 | 93.62 |
| LambdaResNet50 | 16 | 4 | 4 | 7 | 13 | 93.74 (70 epochs) |
Usage
import torch
from model import LambdaConv, LambdaResNet50, LambdaResNet152
x = torch.randn([2, 3, 32, 32])
conv = LambdaConv(3, 128)
print(conv(x).size()) # [2, 128, 32, 32]
# reference
# https://discuss.pytorch.org/t/how-do-i-check-the-number-of-parameters-of-a-model/4325
def get_n_params(model):
pp=0
for p in list(model.parameters()):
nn=1
for s in list(p.size()):
nn = nn*s
pp += nn
return pp
model = LambdaResNet50()
print(get_n_params(model)) # 14.9M (Ours) / 15M(Paper)
model = LambdaResNet152()
print(get_n_params(model)) # 32.8M (Ours) / 35M (Paper)
Parameters
| Model | k | h | u | m | Params (M), Paper | Params (M), Ours |
|---|
| LambdaResNet50 | 16 | 4 | 1 | 23 | 15.0 | 14.9 |
| LambdaResNet50 | 16 | 4 | 4 | 7 | 16.0 | 16.0 |
| LambdaResNet152 | 16 | 4 | 1 | 23 | 35 | 32.8 |
| LambdaResNet200 | 16 | 4 | 1 | 23 | 42 | 35.29 |
Ablation Parameters
| k | h | u | Params (M), Paper | Params (M), Ours |
|---|
| ResNet baseline | | | 25.6 | 25.5 |
| 8 | 2 | 1 | 14.8 | 15.0 |
| 8 | 16 | 1 | 15.6 | 14.9 |
| 2 | 4 | 1 | 14.7 | 14.6 |
| 4 | 4 | 1 | 14.7 | 14.66 |
| 8 | 4 | 1 | 14.8 | 14.66 |
| 16 | 4 | 1 | 15.0 | 14.99 |
| 32 | 4 | 1 | 15.4 | 15.4 |
| 2 | 8 | 1 | 14.7 | 14.5 |
| 4 | 8 | 1 | 14.7 | 14.57 |
| 8 | 8 | 1 | 14.7 | 14.74 |
| 16 | 8 | 1 | 15.1 | 14.1 |
| 32 | 8 | 1 | 15.7 | 15.76 |
| 8 | 8 | 4 | 15.3 | 15.26 |
| 8 | 8 | 8 | 16.0 | 16.0 |
| 16 | 4 | 4 | 16.0 | 16.0 |