Awesome

Adaptive Auto Attack

"Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack"
Ye Liu, Yaya Cheng, Lianli Gao, Xianglong Liu, Qilong Zhang, Jingkuan Song
CVPR 2022

Code and model weights have been released. We will continue to optimize the code.
https://arxiv.org/abs/2203.05154

A practical evaluation method should be convenient (i.e., parameter-free), efficient (i.e., fewer iterations) and reliable (i.e., approaching the lower bound of robustness). Towards this target, we propose a parameter-free Adaptive Auto Attack (A³) evaluation method. We apply A³ to over 50 widely-used defense models. By consuming much fewer iterations than existing methods, i.e, 1/10 on average (10x speed up), we achieve lower robust accuracy in all cases but one. Notably, we won first place out of 1681 teams in CVPR 2021 White-box Adversarial Attacks on Defense Models competitions with this method. 竞赛中文版入口

News

• [March 2022] We extend the A³ to additional datasets (i.e., MNIST, CIFAR-10, CIFAR-100, ImageNet) and metrics (i.e., Linf and L2).
• [March 2022] The paper is accepted at CVPR 2022!

Practical Adversarial Defenses Evaluation

A practical evaluation method should include several advantages:

• Convenient (i.e., parameter-free)
• Efficient (i.e., fewer iterations)
• Reliable (i.e., approaching the lower bound of robustness)

Towards this target, we propose a parameter-free Adaptive Auto Attack (A³) evaluation method.

Note: To have a new defense method added: please check here

CIFAR-10 - Linf

The robust accuracy is evaluated at eps = 8/255, except for those marked with * for which eps = 0.031, where eps is the maximal Linf-norm allowed for the adversarial perturbations. The eps used is the same set in the original papers.

Note: We will gradually refine the evaluation of the defense models.

Note: ‡ indicates models which exploit additional data for training (e.g. unlabeled data, pre-training).

Note: The “acc” column shows the robust accuracies of different models. The “→” column shows the iteration number of forward propagation (million,10⁶), while the “←” column shows the iteration number of backward propagation(million,10⁶). Notably, the “acc” column of A³ shows the difference between the robust accuracies of AA and A³, the “→” and “←” columns of A³ show the speedup factors of A³ relative to AA.

CIFAR-100 - Linf

The robust accuracy is computed at eps = 8/255 in the Linf-norm, except for the models marked with * for which eps = 0.031 is used.

Note: We will gradually refine the evaluation of the defense models.

Note: ‡ indicates models which exploit additional data for training (e.g. unlabeled data, pre-training).

Note: The “acc” column shows the robust accuracies of different models. The “acc” column shows the robust accuracies of different models. The “→” column shows the iteration number of forward propagation (million), while the “←” column shows the iteration number of backward propagation(million). Notably, the “acc” column of A³ shows the difference between the robust accuracies of AA and A³, the “→” and “←” columns of A³ show the speedup factors of A³ relative to AA.

MNIST - Linf

The robust accuracy is computed at eps = 0.3 in the Linf-norm.

Note: We will gradually refine the evaluation of the defense models.

Note: The “acc” column shows the robust accuracies of different models. The “→” column shows the iteration number of forward propagation (million), while the “←” column shows the iteration number of backward propagation(million). Notably, the “acc” column of A³ shows the difference between the robust accuracies of AA and A³, the “→” and “←” columns of A³ show the speedup factors of A³ relative to AA.

CIFAR-10 - L2

The robust accuracy is computed at eps = 0.5 in the L2-norm.

Note: We will gradually refine the evaluation of the defense models.

Note: ‡ indicates models which exploit additional data for training (e.g. unlabeled data, pre-training).

Note: The “acc” column shows the robust accuracies of different models. The “→” column shows the iteration number of forward propagation (million), while the “←” column shows the iteration number of backward propagation(million). Notably, the “acc” column of A³ shows the difference between the robust accuracies of AA and A³, the “→” and “←” columns of A³ show the speedup factors of A³ relative to AA.

ImageNet - Linf

The robust accuracy is computed at eps = 4/255 in the Linf-norm.

Note: We will gradually refine the evaluation of the defense models.

Note: The “acc” column shows the robust accuracies of different models. The “→” column shows the iteration number of forward propagation (million), while the “←” column shows the iteration number of backward propagation(million). Notably, the “acc” column of A³ shows the difference between the robust accuracies of AA and A³, the “→” and “←” columns of A³ show the speedup factors of A³ relative to AA.

ImageNet - L2

The robust accuracy is computed at eps = 3.0 in the L2-norm.

Note: We will gradually refine the evaluation of the defense models.

Note: The “acc” column shows the robust accuracies of different models. The “→” column shows the iteration number of forward propagation (million), while the “←” column shows the iteration number of backward propagation(million). Notably, the “acc” column of A³ shows the difference between the robust accuracies of AA and A³, the “→” and “←” columns of A³ show the speedup factors of A³ relative to AA.

How to use Adaptive Auto Attack

1. Installing dependency packages

pip install -r requirements.txt

2. Download testsets, decompress it, and put it in the "data/" directory.

• Because test sets are loaded differently, you first need to download testsets from the following link (including MNIST, CIFAR-10, CIFAR-100, a subset of ImageNet)：https://drive.google.com/drive/folders/1rM0pRKB2GWY1CZ8wzpymhmWKiON0wYLV?usp=sharing

3. Download defense models you need to test, and put it in the "model_weights/" directory.

• Download models from Baidu online disk: https://pan.baidu.com/s/1_pKv2OBGplSvoNNYdYBJgA
  Extraction Code: arej

4. Run "Adaptive_Auto_Attack_main.py"

Note: The default batch size is running on RTX 3090 (24GB), if the graphics card memory is small, please adjust the batch size.

Demo

1. Installing dependency packages
   pip install -r requirements.txt
2. Run "Adaptive_Auto_Attack_main.py"
   Then you will get the result of TRADES MINIST

Citation

@article{Ye2022Practical,
author = {Ye Liu and Yaya Cheng and Lianli Gao and Xianglong Liu and Qilong Zhang and Jingkuan Song},
title = {Practical Evaluation of Adversarial Robustness via Adaptive Auto Attack},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
year = {2022},
eprint = {arXiv:2203.05154},
}