Home

Awesome

:sparkles: GOOD: A Graph Out-of-Distribution Benchmark :sparkles:

Documentation Status Last Commit License codecov CircleCI GOOD stars

Documentation | Paper (and Leaderboard)

<!-- > We are actively building the document. --> <!-- [**GOOD: A Graph Out-of-Distribution Benchmark.**](https://arxiv.org/abs/2206.08452) Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. --> <!-- :fire:**New! The GOOD is now also parts of the software library [DIG](https://github.com/divelab/DIG)! If you wish to use the GOOD datasets with DIG features, you can directly use the [DIG](https://github.com/divelab/DIG) library!** -->

:tada: We are glad to announce that our GOOD paper is accepted by NeurIPS 2022 Datasets and Benchmarks Track! :confetti_ball:

:eye:Note that we split the licences of our code and datasets, please refer to different licenses for different usages.

Roadmap

Tutorial

Algorithms (plan to include)

New features

Table of contents

Overview

GOOD (Graph OOD) is a graph out-of-distribution (OOD) algorithm benchmarking library depending on PyTorch and PyG to make develop and benchmark OOD algorithms easily.

Currently, GOOD contains 11 datasets with 17 domain selections. When combined with covariate, concept, and no shifts, we obtain 51 different splits. We provide performance results on 10 commonly used baseline methods (ERM, IRM, VREx, GroupDRO, Coral, DANN, MixupForGraph, DIR, EERM,SRGNN) including 4 graph specific methods with 10 random runs.

The GOOD dataset summaries are shown in the following figure.

Dataset

Why GOOD?

Whether you are an experienced researcher of graph out-of-distribution problems or a first-time learner of graph deep learning, here are several reasons to use GOOD as your Graph OOD research, study, and development toolkit.

Installation

Conda dependencies

GOOD depends on PyTorch (>=1.6.0), PyG (>=2.0), and RDKit (>=2020.09.5). For more details: conda environment

Note that we currently test on PyTorch (==1.10.1), PyG (==2.0.3), RDKit (==2020.09.5); thus we strongly encourage to install these versions.

Attention! Due to a known issue, please install PyG through Pip to avoid incompatibility.

Pip

Installation for only modules

pip install graph-ood

Installation for Project usages (recommended)

git clone https://github.com/divelab/GOOD.git && cd GOOD
pip install -e .

Quick Tutorial

Module usage

GOOD datasets

There are two ways to import 8 GOOD datasets with 14 domain selections and a total 42 splits, but for simplicity, we only show one of them. Please refer to Tutorial for more details.

# Directly import
from GOOD.data.good_datasets.good_hiv import GOODHIV
hiv_datasets, hiv_meta_info = GOODHIV.load(dataset_root, domain='scaffold', shift='covariate', generate=False)

GOOD GNNs

The best and fair way to compare algorithms with the leaderboard is to use the same and similar graph encoder structure; therefore, we provide GOOD GNN APIs to support. Here, we use an objectified dictionary config to pass parameters. More details about the config: Documents of config In this case, we recommend using this package for project usages.

To use exact GNN

from GOOD.networks.models.GCNs import GCN
model = GCN(config)

To only use parts of GNN

from GOOD.networks.models.GINvirtualnode import GINEncoder
encoder = GINEncoder(config)

GOOD algorithms

Try to apply OOD algorithms to your own models?

from GOOD.ood_algorithms.algorithms.VREx import VREx
ood_algorithm = VREx(config)
# Then you can provide it to your model for necessary ood parameters, 
# and use its hook-like function to process your input, output, and loss.

Project usage

It is a good beginning to make it work directly. Here, we provide the command line script goodtg (GOOD to go) to access the main function located at GOOD.kernel.pipeline:main. Choosing a config file in configs/GOOD_configs, we can start a task:

goodtg --config_path GOOD_configs/GOODCMNIST/color/concept/DANN.yaml

Specifically, the task is clearly divided into three steps:

  1. Config
from GOOD import config_summoner
from GOOD.utils.args import args_parser
from GOOD.utils.logger import load_logger
args = args_parser()
config = config_summoner(args)
load_logger(config)
  1. Loader
from GOOD.kernel.pipeline import initialize_model_dataset
from GOOD.ood_algorithms.ood_manager import load_ood_alg
model, loader = initialize_model_dataset(config)
ood_algorithm = load_ood_alg(config.ood.ood_alg, config)
  1. Train/test pipeline
from GOOD.kernel.pipeline import load_task
load_task(config.task, model, loader, ood_algorithm, config)

Please refer to Tutorial for more details.

Reproducibility

For reproducibility, we provide full configurations used to obtain leaderboard results in configs/GOOD_configs.

We further provide two tests: dataset regeneration test and test result check.

Dataset regeneration test

This test regenerates all datasets again and compares them with the datasets used in the original training process locates. Test details can be found at test_regenerate_datasets.py. For a quick review, we provide a full regeneration test report.

Leaderboard results test

This test loads all checkpoints in round 1 and compares their results with saved ones. Test details can be found at test_reproduce_round1.py. For a quick review, we also post our full round1 reproduce report.

These reports are in html format. Please download them and open them in your browser.: )

Training plots: The training plots for all algorithms in round 1 can be found HERE.

Sampled tests

In order to keep the validity of our code all the time, we link our project with circleci service and provide several sampled tests to go through (because of the limitation of computational resources in CI platforms).

Citing GOOD

If you find this repository helpful, please cite our paper.

@inproceedings{
gui2022good,
title={{GOOD}: A Graph Out-of-Distribution Benchmark},
author={Shurui Gui and Xiner Li and Limei Wang and Shuiwang Ji},
booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
year={2022},
url={https://openreview.net/forum?id=8hHg-zs_p-h}
}

License

The GOOD datasets are under MIT license. The GOOD code are under GPLv3 license.

Discussion

Please submit new issues or start a new discussion for any technical or other questions.

Contact

Please feel free to contact Shurui Gui, Xiner Li, or Shuiwang Ji!