Home

Awesome

EasyRobust

<div align="center">

license open issues GitHub pull-requests GitHub latest release

</div>

What's New

Our Research Project

Introduction

EasyRobust is an Easy-to-use library for state-of-the-art Robust Computer Vision Research with PyTorch. EasyRobust aims to accelerate research cycle in robust vision, by collecting comprehensive robust training techniques and benchmarking them with various robustness metrics. The key features includes:

Technical Articles

We have a series of technical articles on the functionalities of EasyRobust.

Installation

Install from Source:

$ git clone https://github.com/alibaba/easyrobust.git
$ cd easyrobust
$ pip install -e .

Install from PyPI:

$ pip install easyrobust

download the ImageNet dataset and place into /path/to/imagenet. Specify $ImageNetDataDir as ImageNet path by:

$ export ImageNetDataDir=/path/to/imagenet

[Optional]: If you use EasyRobust to evaluate the model robustness, download the benchmark dataset by:

$ sh download_data.sh

[Optional]: If you use analysis tools in tools/, install extra requirements by:

$ pip install -r requirements/optional.txt

Docker

We have provided a runnable environment in docker/Dockerfile for users who do not want to install by pip. To use it, please confirm that docker and nvidia-docker have installed. Then run the following command:

docker build -t alibaba/easyrobust:v1 -f docker/Dockerfile . 

Getting Started

EasyRobust focuses on the basic usages of: (1) Evaluate and benchmark the robustness of a pretrained models and (2) Train your own robust models or reproduce the results of previous SOTA methods.

1. How to evaluate and benchmark the robustness of given models?

It only requires a few lines to evaluate the robustness of a model using EasyRobust. We give a minimalist example in benchmarks/resnet50_example.py:

#############################################################
#         Define your model
#############################################################
model = torchvision.models.resnet50(pretrained=True)
model = model.eval()
if torch.cuda.is_available(): model = model.cuda()

#############################################################
#         Start Evaluation
#############################################################

# ood
evaluate_imagenet_val(model, 'benchmarks/data/imagenet-val')
evaluate_imagenet_a(model, 'benchmarks/data/imagenet-a')
evaluate_imagenet_r(model, 'benchmarks/data/imagenet-r')
evaluate_imagenet_sketch(model, 'benchmarks/data/imagenet-sketch')
evaluate_imagenet_v2(model, 'benchmarks/data/imagenetv2')
evaluate_stylized_imagenet(model, 'benchmarks/data/imagenet-style')
evaluate_imagenet_c(model, 'benchmarks/data/imagenet-c')
# objectnet is optional since it spends a lot of disk storage. we skip it here. 
# evaluate_objectnet(model, 'benchmarks/data/ObjectNet/images')

# adversarial
evaluate_imagenet_autoattack(model, 'benchmarks/data/imagenet-val')

You can do evaluation by simply running the command: python benchmarks/resnet50_example.py. After running is completed, your will get the following output:

Top1 Accuracy on the ImageNet-Val: 76.1%
Top1 Accuracy on the ImageNet-A: 0.0%
Top1 Accuracy on the ImageNet-R: 36.2%
Top1 Accuracy on the ImageNet-Sketch: 24.1%
Top1 Accuracy on the ImageNet-V2: 63.2%
Top1 Accuracy on the Stylized-ImageNet: 7.4%
Top1 accuracy 39.2%, mCE: 76.7 on the ImageNet-C
Top1 Accuracy on the AutoAttack: 0.0%

2. How to use EasyRobust to train my own robust models?

We implement most robust training methods in the folder examples/imageclassification/imagenet/. All of them are based on a basic training script: examples/imageclassification/imagenet/base_training_script.py. By comparing the difference, you can clearly see where and which hyperparameters of basic training are modified to create a robust training example. Below we present the tutorials of some classic methods:

Analytical Tools

see tools/README.md

Model Zoo and Baselines

Submit your models

We provide a tool benchmarks/benchmark.py to help users directly benchmark their models:

Usage: 
    python benchmarks/benchmark.py [OPTIONS...]

OPTIONS:
    --model [ARCH in timm]
    --data_dir [PATH of the bencmark datasets]
    --ckpt_path [URL or PATH of the model weights]

If you are willing to submit the model to our benchmarks, you can prepare a python script similar to benchmarks/benchmark.py and weights file xxx.pth, zip all the files. Then open an issue with the "Submit Model" template and provide a json storing submit information. Below is a submission template in adversarial robustness benchmark of image classification:

## Submit Json Information

{"date": "19/06/2017", 
 "extra_data": "no", 
 "model": "<b>Adversarial Training</b>", 
 "institution": "MIT", 
 "paper_link": "https://arxiv.org/abs/1706.06083", 
 "code_link": "", 
 "architecture": "swin-b", 
 "training framework": "easyrobust (v1)", 
 "ImageNet-val": 75.05, 
 "autoattack": 47.42, 
 "files": "<a href=http://alisec-competition.oss-cn-shanghai.aliyuncs.com/xiaofeng/imagenet_pretrained_models/advtrain_models/advtrain_swin_base_patch4_window7_224_ep4.pth >download</a>", 
 "advrob_imgcls_leaderboard": true, 
 "oodrob_imgcls_leaderboard": false, 
 "advrob_objdet_leaderboard": false, 
 "oodrob_objdet_leaderboard": false}

We will check the result and present your result into the benchmark if there is no problem. For submission template of other benchmarks, check submit-model.md.

Below is the model zoo and benchmark of the EasyRobust. All the results are runned by benchmarks/adv_robust_bench.sh and benchmarks/non_adv_robust_bench.sh.

Adversarial Robust Benchmark (sorted by AutoAttack)

Training FrameworkMethodModelImageNet-ValAutoAttackFiles
EasyRobust (V1)Adversarial TrainingSwin-B75.05%47.42%ckpt
EasyRobust (V1)Adversarial TrainingSwin-S73.41%46.76%ckpt
EasyRobust (V1)Adversarial TrainingViT-B/1670.64%43.04%ckpt
EasyRobust (V1)Adversarial TrainingEfficientNet-B367.65%41.72%ckpt
EasyRobust (V1)Adversarial TrainingResNet10169.51%41.04%ckpt
EasyRobust (V1)Adversarial TrainingViT-S/1666.43%39.20%ckpt
EasyRobust (V1)Adversarial TrainingEfficientNet-B264.75%38.54%ckpt
EasyRobust (V1)Adversarial TrainingResNeSt50d70.03%38.52%ckpt
EasyRobust (V1)Adversarial TrainingViT-B/3265.58%37.38%ckpt
EasyRobust (V1)Adversarial TrainingEfficientNet-B163.99%37.20%ckpt
EasyRobust (V1)Adversarial TrainingSEResNet10171.11%37.18%ckpt
EasyRobust (V1)Adversarial TrainingResNeXt50_32x4d67.39%36.42%ckpt
EasyRobust (V1)Adversarial TrainingEfficientNet-B061.83%35.06%ckpt
robustnessAdversarial TrainingResNet5064.02%34.96%ckpt
EasyRobust (Ours)Adversarial TrainingResNet5065.1%34.9%ckpt/args/logs
EasyRobust (V1)Adversarial TrainingSEResNet5066.68%33.56%ckpt
EasyRobust (V1)Adversarial TrainingDenseNet12160.90%29.78%ckpt
OfficialFree ATResNet5059.96%28.58%ckpt
OfficialFGSM ATResNet5055.62%26.24%ckpt
EasyRobust (V1)Adversarial TrainingVGG1659.96%25.92%ckpt

Non-Adversarial Robust Benchmark (sorted by ImageNet-C)

Training FrameworkMethodModelFilesImageNet-ValV2C (mCE↓)RASketchStylizedObjectNet
EasyRobust (Ours)DATViT-B/16ckpt/args/logs81.38%69.99%45.5949.64%24.61%36.46%24.84%20.12%
EasyRobust (Ours)-RVT-S*ckpt/args/logs82.10%71.40%48.2247.84%26.93%35.34%20.71%23.24%
Official-RVT-S*ckpt81.82%71.05%49.4247.33%26.53%34.22%20.48%23.11%
EasyRobust (Ours)-DrViT-Sckpt/args/logs80.66%69.62%49.9643.68%20.79%31.13%17.89%20.50%
--DrViT-S-77.03%64.49%56.8939.02%11.85%28.78%14.22%26.49%
OfficialPRIMEResNet50ckpt76.91%65.42%57.4942.20%2.21%29.82%13.94%16.59%
EasyRobust (Ours)PRIMEResNet50ckpt/args/logs76.64%64.37%57.6241.95%2.07%29.63%13.56%16.28%
EasyRobust (Ours)DeepAugmentResNet50ckpt/args/logs76.58%64.77%60.2742.80%3.62%29.65%14.88%16.88%
OfficialDeepAugmentResNet50ckpt76.66%65.24%60.3742.17%3.46%29.50%14.68%17.13%
EasyRobust (Ours)AugmixResNet50ckpt/args/logs77.81%65.60%64.1443.34%4.04%29.81%12.33%17.21%
EasyRobust (Ours)APRResNet50ckpt/args/logs76.28%64.78%64.8942.17%4.18%28.90%13.03%16.78%
OfficialAugmixResNet50ckpt77.54%65.42%65.2741.04%3.78%28.48%11.24%17.54%
OfficialAPRResNet50ckpt75.61%64.24%65.5641.35%3.20%28.37%13.01%16.61%
OfficialS&T DebiasedResNet50ckpt76.91%65.04%67.5540.81%3.50%28.41%17.40%17.38%
EasyRobust (Ours)SIN+INResNet50ckpt/args/logs75.46%63.50%67.7342.34%2.47%31.39%59.37%16.17%
OfficialSIN+INResNet50ckpt74.59%62.43%69.3241.45%1.95%29.69%57.38%15.93%
Non-OfficialAdvPropResNet50ckpt77.04%65.27%70.8140.13%3.45%25.95%10.01%18.23%
EasyRobust (Ours)S&T DebiasedResNet50ckpt/args/logs77.21%65.10%70.9838.59%3.28%26.09%14.59%16.99%
EasyRobust (Ours)AdvPropResNet50ckpt/args/logs76.64%64.35%77.6437.43%2.83%24.71%7.33%16.82%

Credits

EasyRobust concretizes previous excellent works by many different authors. We'd like to thank, in particular, the following implementations which have helped us in our development:

Citing EasyRobust

We provide a BibTeX entry for users who apply EasyRobust to help their research:

@misc{mao2022easyrobust,
  author =       {Xiaofeng Mao and Yuefeng Chen and Xiaodan Li and Gege Qi and Ranjie Duan and Rong Zhang and Hui Xue},
  title =        {EasyRobust: A Comprehensive and Easy-to-use Toolkit for Robust Computer Vision},
  howpublished = {\url{https://github.com/alibaba/easyrobust}},
  year =         {2022}
}