Awesome

OnlineLabelSmoothing

This is a re-implementation of Online Label Smoothing. The code is written based on my understanding of the paper. If there's any bug in my code, please tell me in the Issues page.

Usage

from OLS import OnlineLabelSmoothing

ols_loss = OnlineLabelSmoothing(num_classes=1000, use_gpu=True)

# Training
for epoch in range(total_epoch):
    # train()
    # test()
    ols_loss.update()

# Saving
torch.save({'ols': ols_loss.matrix.cpu().data}, 'ols.pth')

Results

Environment

• Python 3.7
• PyTorch 1.6.0
• GPU: Tesla V100 32GB * 1

Other Setting

num_classes: 1000
optimizer: SGD
init_lr: 0.1
weight_decay: 0.0001
momentum: 0.9
lr_gamma: 0.1
total_epoch: 250
batch_size: 256
num_workers: 20
random_seed: 2020
amp: True # automatic mixed-precision training, this function is offered by pytorch

Train

• use single gpu

python train.py --amp -s cos --loss ce ols --loss_w 0.5 0.5

• use multi gpus single node

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch\
--nproc_per_node=2 --master_addr 127.0.0.7 --master_port 23456\
train.py --multi-gpus 1 -nw 20 --amp -s multi --loss ce ols --loss_w 0.5 0.5

• use multi gpus multi nodes

# Limited computing resources

Accuracy on Validation Set of ImageNet2012

Although I used AMP(automatic mixed-precision) to speed up my training, it still took me nearly five days, so I didn't do any other experiments with ols. But there are other records of training ImageNet in my blog.

Reference

• Delving Deep into Label Smoothing
• https://github.com/zhangchbin/OnlineLabelSmooth