Awesome

GraSH: Successive Halving for Knowledge Graphs

This is the code and configuration accompanying the paper "Start Small, Think Big: On Hyperparameter Optimization for Large-Scale Knowledge Graph Embeddings" presented at ECML-PKDD 2022. The code extends the knowledge graph embedding library for distributed training Dist-KGE. For documentation on Dist-KGE refer to the Dist-KGE repository. We provide the hyperparameter settings for the searches and finally selected trials in /examples/experiments/.

UPDATE: GraSH was recently merged into our main library LibKGE. All configs from this repository, except the ones for Freebase that require distributed training, can be executed in LibKGE. Please use LibKGE for your own experiments with GraSH.

Table of contents

1. Quick start
2. Configuration of GraSH Search
3. Run a GraSH hyperparameter search
4. Results and Configurations
5. How to cite

Quick start

Setup

# retrieve and install project in development mode
git clone https://github.com/uma-pi1/grash.git
cd grash
pip install -e .

# download and preprocess datasets
cd data
sh download_all.sh
cd ..

Training

# train an example model on a toy dataset (you can omit '--job.device cpu' when you have a gpu)
python -m kge start examples/toy-complex-train.yaml --job.device cpu

This example will train on a toy dataset in a sequential setup on CPU

GraSH Hyperparameter Search

# perform a search with GraSH on a toy dataset (you can omit '--job.device cpu' when you have a gpu)
python -m kge start examples/toy-complex-search-grash.yaml --job.device cpu

This example will perform a small GraSH search with 16 trials on a toy dataset in a sequential setup on CPU

Configuration of GraSH Search

The most important configuration options for a hyperparameter search with GraSH are:

dataset:
  name: yago3-10
grash_search:
  eta: 4
  num_trials: 64
  search_budget: 3
  variant: combined
  parameters: # define your search space here
job:
  type: search
model: complex
train:
  max_epochs: 400

• eta defines the reduction factor during the search. Per round the number of remaining trials is reduced to 1/eta
• search_budget is defined in "number of full training runs". The default choice search_budget=3, for example, corresponds to an overall search cost of three full training runs.
• variant controls which reduction technique to use (only epoch, only graph, or combined)

Run a GraSH hyperparameter search

Run the default search on yago3-10 with the following command:

python -m kge start examples/experiments/search_configs/yago3-10/search-complex-yago-combined.yaml

The k-core subgraphs will automatically be generated and saved to data/yago3-10/subsets/k-core/. By default, each experiment will create a new folder in local/experiments/<timestamp>-<config-name> where the results can be found.

Results and Configurations

All results were obtained with the GraSH default settings (num_trials=64, eta=4, search_budget=3, variant=combined)

Yago3-10

Wikidata5M

Freebase

How to cite

@inproceedings{kochsiek2022start,
  title={Start Small, Think Big: On Hyperparameter Optimization for Large-Scale Knowledge Graph Embeddings},
  author={Kochsiek, Adrian and Niesel, Fritz and Gemulla, Rainer},
  booktitle={Joint European Conference on Machine Learning and Knowledge Discovery in Databases},
  year={2022}
}