Awesome
Understanding Adversarial Flow Robustness
PyTorch implementation of Towards Understanding Adversarial Robustness of Optical Flow Networks.
If this work is useful to you, please consider citing our paper:
@inproceedings{schrodi2022towards,
title={Towards Understanding Adversarial Robustness of Optical Flow Networks},
author={Schrodi, Simon and Saikia, Tonmoy and Brox, Thomas},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
year={2022}
}
Prerequisites
To setup your Python environment follow the steps below:
-
Clone this repository.
-
cd
into this repository. -
Create a conda environment:
conda create --name flow_rob --file conda_requirements.txt
or if your system uses GCC-7 as standard compiler, you can just run
conda create --name flow_rob python=3.8
-
Activate the conda environment
conda activate flow_rob
. -
Run
conda develop "${pwd}"
(if you are in the repository) orconda develop /path/to/project/understanding_flow_robustness
. -
Install PyTorch 1.4.0 (Poetry cannot properly deal with that). Be aware that the CUDA version and GCC version need to match; see here for more information.
bash setup/install_torch.sh
- Install poetry:
bash setup/install_poetry.sh
or follow the installation instructions here.
-
Run
poetry install
. -
Install custom cuda layers (requires CUDA runtime!):
bash setup/install_flownet2_deps.sh
- Install Spatial correlation sampler:
pip install spatial-correlation-sampler
If you have problems installing it, please refer to here. Note that we need to re-compile this package if run on different GPU_CCs. There might be soon a better solution, see here. In the meantime, you need to run the following command everytime (unless you use the same GPU_CC):
pip uninstall spatial-correlation-sampler -y
cd models/Pytorch-Correlation-extension
rm -rf *_cuda.egg-info build dist __pycache__
python setup.py install
cd ../..
- (Optional) Run
pre-commit install
.
Preparing the data
- By default, we assume the following folder structure:
├── datasets
├── KITTI
├── 2012_prepared
├── 2011_09_26_drive_0001_sync_02
├── 2011_09_26_drive_0001_sync_03
├── ...
├── train.txt
├── val.txt
├── 2012
├── test
├── training
├── 2015
├── test
├── training
├── FlyingChairs_release
├── data
├── FlyingThings3D
├── frames_cleanpass
├── frames_finalpass
├── optical_flow
├── Sintel
├── test
├── training
-
Download the following datasets
- The raw KITTI dataset with this script from the official website,
- KITTI 2012 (stereo flow),
- KITTI 2015 (scene 4low),
- Flying Chairs,
- Flying Things 3D, and
- Sintel.
-
Run
python3 -m setup.prepare_train_data.py /path/to/KITTI/raw --dataset-format 'kitti' --dump-root /path/to/KITTI/2012_prepared --width 1280 --height 384 --num-threads 1 --with-gt
- Create symbolic links (if not saved there already):
mkdir datasets
mkdir datasets/KITTI
ln -s /path/to/KITTI/2012_prepared datasets/KITTI/2012_prepared
ln -s /path/to/KITTI/2012/ datasets/KITTI/2012
ln -s /path/to/KITTI/2015/ datasets/KITTI/2015
ln -s /path/to/FlyingChairs_release/ datasets/FlyingChairs_release
ln -s /path/to/FlyingThings3D/ datasets/FlyingThings3D
ln -s /path/to/Sintel/ datasets/Sintel
- Run
python3 setup/setup_dataset_kitti.py
Pretrained models
- We assume the following folder structure:
├── pretrained_models
├── spynet_models
├── adv_kitti2012_pwcnet_ifgsm_l2_0.02.pth
├── adv_kitti2012_raft_ifgsm_l2_0.02.pth
├── adv_kitti2012_robustFlow_ifgsm_l2_0.02.pth
├── FlowNet2_checkpoint.pth.tar
├── FlowNet2-C_checkpoint.pth.tar
├── FlowNet2-S_checkpoint.pth.tar
├── larger_field_3x3_x0_l2.pth
├── pwc_net_chairs.pth.tar
├── raft_flowNetCEnc_noSeparateContext.pth
├── raft-things.pth
├── RobustFlowNetC.pth
- Download the pretrained models for FlowNetC, FlowNetS, FlowNet2, PWC-Net, RAFT, Robust FlowNetC, FlowNetC trained with our training schedule, adv. trained Robust FlowNetC, RAFT with FlowNetC encoder and no separate context encoder, adv. trained PWC-Net, and adv. trained RAFT. Note that you have to transform the RAFT models to be compatible with PyTorch 1.4; see here for more information. For SPyNet copy the weights from here to
pretrained_models/spynet_models
.
For convenience, you can run bash setup/download_weights
to download all model weights.
- Create symbolic links (if not saved there already):
mkdir pretrained_models
ln -s /path/to/FlowNet2-C_checkpoint.pth.tar pretrained_models/FlowNet2-C_checkpoint.pth.tar
ln -s /path/to/FlowNet2-S_checkpoint.pth.tar pretrained_models/FlowNet2-S_checkpoint.pth.tar
ln -s /path/to/FlowNet2_checkpoint.pth.tar pretrained_models/FlowNet2_checkpoint.pth.tar
ln -s /path/to/pwc_net_chairs.pth.tar pretrained_models/pwc_net_chairs.pth.tar
ln -s /path/to/raft-things.pth pretrained_models/raft-things.pth
Patch-based Adversarial Attacks on Optical Flow
Generating Adversarial Patches
To generate adversarial patches, run
python3 patch_attacks/main.py \
--name $path \
--valset kitti2012 \
--workers 1 \
--flownet $flownet \
--lr 1000 \
--epochs 40 \
--patch-size 0.1329 \
--seed 42
Analysis & Evaluation
To test the patch across various locations and find the worst one (Section 4.1), run
python3 patch_attacks/test_moving_patch.py \
--name $path \
--instance $instance \
--patch_name $name \
--flownet $flownet \
--stride 25 \
--norotate
To reproduce our experiments in Table 2 (requires previous generation of an adversarial patch), run
python3 patch_attacks/test_replace_features.py \
--name $path \
--feature_keys corr conv_redir \
--flownet FlowNetC \
--instance $instance \
--patch_name $patch_name
python3 patch_attacks/test_replace_features.py \
--name $path \
--feature_keys corr \
--flownet FlowNetC \
--instance $instance \
--patch_name $patch_name
python3 patch_attacks/test_replace_features.py \
--name $path \
--feature_keys conv_redir \
--flownet FlowNetC \
--instance $instance \
--patch_name $patch_name
To create the t-SNE embeddings in Figure 3 and 9, run
python3 patch_attacks/test_patch_embeddings.py \
--name $path \
--instance $instance \
--patch_name $patch_name \
--flownet $flownet
Handcrafted Patch Attacks
To attack without the need of any optimization (Section 5), run
python3 patch_attacks/test_moving_patch.py \
--name $path \
--self_correlated_patch vstripes \
--patch_size $patch_size \
--flownet $flownet \
--stride 25 \
--norotate
You can run the other scripts for analysis and evaluation accordingly.
Global Adversarial Attacks on Optical Flow
Attacking & Evaluation
To reproduce the experiments for untargeted global white-box attacks (Supplemental Section H), run
python3 global_attacks/run_perturb_model.py \
--output_path $output_path \
--dataset kitti2015 \
--n_height 256 \
--n_width 640 \
--output_norm $output_norm \
--learning_rate $lr \
--n_step $n_step \
--perturb_method $perturb_method \
--flow_loss $flow_loss
--flownet $flownet \
--write_out \
To reproduce the results for the targeted 42 attack (Figure 11), run
python3 global_attacks/run_perturb_model.py \
--output_path $output_path \
--dataset kitti2015 \
--n_height 256 \
--n_width 640 \
--output_norm $output_norm \
--learning_rate $lr \
--perturb_method $perturb_method \
--flownet $flownet \
--write_out \
--targeted \
--arbitrary_gt_index fun
or to attack with an arbitrary GT from KITTI 2015 (Supplemental Section I), run
python3 global_attacks/run_perturb_model.py \
--output_path /misc/lmbraid19/schrodi/stereopagnosia \
--dataset kitti2015 \
--n_height 256 \
--n_width 640 \
--output_norm $output_norm \
--learning_rate $lr \
--n_step $n_step \
--perturb_method $perturb_method \
--flownet $flownet \
--seed 0 \
--write_out \
--flow_loss $flow_loss \
--targeted
--arbitrary_gt_index $arbitrary_index \
where arbitrary_index
can be between 0 and 199.
To generate universal adversarial perturbation (Section 7), run
python3 global_attacks/run_perturb_model.py \
--output_path $output_path \
--dataset kitti2015 \
--n_height 256 \
--n_width 640 \
--perturb_method $perturb_method \
--flownet $flownet \
--perturb_mode same \
--write_out
and then to attack, run
python3 $WORKDIR/global_attacks/run_perturb_model.py \
--output_path $output_path \
--dataset kitti2015 \
--flownet $flownet \
--universal_evaluation \
--output_norm $output_norm \
--perturb_method $perturb_method \
--flow_loss $flow_loss \
--folder_name $folder_name \
--epoch_number $epoch_number \
--n_height 256 \
--n_width 640 \
--write_out
To apply common image corruptions and test flow nets, run
python3 global_attacks/run_perturb_model.py \
--output_path $output_path \
--dataset kitti2015 \
--n_height 384 \
--n_width 1280 \
--perturb_method $perturb_method \
--flownet $flownet \
--perturb_mode same \
--write_out
Adversarial Training
To train adversarially robust flow nets (Supplemental Section K), run:
python training/train.py --name $name \
--restore_ckpt pretrained_models/raft-things.pth \
--adv_train \
--stage kitti2012 \
--ckpt_dir $ckpt_dir \
--gpus 0 1 \
--num_steps $num_steps \
--batch_size 1 \
--lr 0.000125 \
--image_size 256 640 \
--wdecay 0.0001 \
--flownet $flownet \
--perturb_method $perturb_method \
--output_norm $output_norm \
--perturb_learning_rate $perturb_learning_rate \
--perturb_n_step $perturb_n_step \
--flow_loss $flow_loss
Acknowledgements
We thank several GitHub users for their contributions which are used in this repository:
- Correlation module from ClementPinard/Pytorch-Correlation-extension.
- The basis for patch-based attacks is taken from anuragranj/flowattack.
- The basis for global attacks is taken from alexklwong/stereopagnosia.
- The basis for training of flow networks is taken from princeton-vl/RAFT.