Home

Awesome

GRB

PyPi Latest Release Documentation Status License

Homepage | Paper | Datasets | Leaderboard | Documentation

Graph Robustness Benchmark (GRB) provides scalable, unified, modular, and reproducible evaluation on the adversarial robustness of graph machine learning models. GRB has elaborated datasets, unified evaluation pipeline, modular coding framework, and reproducible leaderboards, which facilitate the developments of graph adversarial learning, summarizing existing progress and generating insights into future research.

<div align=center><img width="700" src=https://github.com/THUDM/grb/blob/master/docs/source/_static/grb_key_features.png/></div>

Updates

Get Started

Installation

Install grb via pip (current version v0.1.0):

pip install grb

Install grb via git (for the newest version):

git clone git@github.com:THUDM/grb.git
cd grb
pip install -e .

Preparation

GRB provides all necessary components to ensure the reproducibility of evaluation results. Get datasets from link or download them by running the following script:

cd ./scripts
sh download_dataset.sh

Get attack results (adversarial adjacency matrix and features) from link or download them by running the following script:

sh download_attack_results.sh

Get saved models (model weights) from link or download them by running the following script:

sh download_saved_models.sh

Usage of GRB Modules

Training a GML model

An example of training Graph Convolutional Network (GCN) on grb-cora dataset.

import torch  # pytorch backend
from grb.dataset import Dataset
from grb.model.torch import GCN
from grb.trainer.trainer import Trainer

# Load data
dataset = Dataset(name='grb-cora', mode='easy',
                  feat_norm='arctan')
# Build model
model = GCN(in_features=dataset.num_features,
            out_features=dataset.num_classes,
            hidden_features=[64, 64])
# Training
adam = torch.optim.Adam(model.parameters(), lr=0.01)
trainer = Trainer(dataset=dataset, optimizer=adam,
                  loss=torch.nn.functional.nll_loss)
trainer.train(model=model, n_epoch=200, dropout=0.5,
              train_mode='inductive')

Adversarial attack

An example of applying Topological Defective Graph Injection Attack (TDGIA) on trained GCN model.

from grb.attack.injection.tdgia import TDGIA

# Attack configuration
tdgia = TDGIA(lr=0.01, 
              n_epoch=10,
              n_inject_max=20, 
              n_edge_max=20,
              feat_lim_min=-0.9, 
              feat_lim_max=0.9,
              sequential_step=0.2)
# Apply attack
rst = tdgia.attack(model=model,
                   adj=dataset.adj,
                   features=dataset.features,
                   target_mask=dataset.test_mask)
# Get modified adj and features
adj_attack, features_attack = rst

GRB Evaluation

Evaluation scenarios (Injection attack as an example)

GRB

GRB provides unified evaluation scenarios for fair comparisons between attacks and defenses. The example scenario is Black-box, Evasion, Inductive, Injection.

GRB Leaderboards

GRB maintains leaderboards that permits a fair comparision across various attacks and defenses. To ensure the reproducibility, we provide all necessary information including datasets, attack results, saved models, etc. Besides, all results on the leaderboards can be easily reproduced by running the following scripts (e.g., [leaderboard for grb-cora dataset](https://cogdl.ai/grb/leaderboard/cora, compatible with v0.1.0)):

sh run_leaderboard_pipeline.sh -d grb-cora -g 0 -s ./leaderboard -n 0
Usage: run_leaderboard_pipeline.sh [-d <string>] [-g <int>] [-s <string>] [-n <int>]
Pipeline for reproducing leaderboard on the chosen dataset.
    -h      Display help message.
    -d      Choose a dataset.
    -s      Set a directory to save leaderboard files.
    -n      Choose the number of an attack from 0 to 9.
    -g      Choose a GPU device. -1 for CPU.

Submission

We welcome researchers to submit new methods including attacks, defenses, or new GML models to enrich the GRB leaderboard. For future submissions, one should follow the GRB Evaluation Rules and respect the reproducibility.

Please submit your methods via the google form GRB submission. Our team will verify the result within a week.

Requirements

Citing GRB

If you find GRB useful for your research, please cite our paper:

@article{zheng2021grb,
  title={Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning},
  author={Zheng, Qinkai and Zou, Xu and Dong, Yuxiao and Cen, Yukuo and Yin, Da and Xu, Jiarong and Yang, Yang and Tang, Jie},
  journal={Neural Information Processing Systems Track on Datasets and Benchmarks 2021},
  year={2021}
}

Contact

In case of any problem, please contact us via email: cogdl.grbteam@gmail.com. We also welcome researchers to join our Google Group for further discussion on the adversarial robustness of graph machine learning.