Home

Awesome

KZSL: Benchmarking Knowledge-driven Zero-shot Learning

1. Introduction

This repository includes resources for benchmarking paper "Benchmarking Knowledge-driven Zero-shot Learning". In this work, we created systemic resources for KG-based ZSL research on zero-shot image classification (ZS-IMGC), zero-shot relation extraction (ZS-RE) and zero-shot knowledge graph (KG) completion (ZS-KGC), including 6 ZSL datasets and their corresponding KGs, with the goal of providing standard benchmarks and ranging semantics settings for studying and comparing different KG-based ZSL methods. The benchmarking study presented in the paper shows the effectiveness and great potential usage of our proposed resources. In the future, we hope this resource can serve as an important cornerstone to promote more advanced ZSL methods and more effective solutions for applying KGs for augmenting machine learning, and build a solid neural-symbolic paradigm for advancing the development of artificial intelligence.

2. Zero-shot Image Classification (ZS-IMGC)

ZS-IMGC aims to predict images with new classes that have no labeled training images. Here, we provide three standard ZS-IMGC datasets, including ImNet-A and ImNet-O constructed by ourselves, and one widely-used benchmark named AwA2. For each dataset, we construct a KG to represent its different kinds of class semantics, including class attribute, text and hierarchy, as well as common sense knowledge from ConceptNet and logical relationships between classes (e.g., disjointness).

Statistics

Dataset# Classes (Total/Seen/Unseen)# Attributes# Images
ImNet-A80 / 28 / 528577,323
ImNet-O35 / 10 / 254039,361
AwA250 / 40 / 108537,322
KG# Entities# Relations# Triples
ImNet-A8,9204110,461
ImNet-O3,148313,990
AwA29,1954214,112

Usage

ZSL Datasets (Class Splits and Image Features)

   See detailed introductions for these files here.

KGs

Each KG is composed of RDF triples and stored in a CSV file with three columns corresponding to subjects, relations and objects. We have provided these KGs in our repository. You can browse them in the folder ZS_IMGC/KG/. Note the CSV file is saved with delimiter '\t'.

3. Zero-shot Relation Extraction (ZS-RE)

ZS-RE aims to predict/extract the unseen relations between two given entity mentions by a sentence. Here, we construct a ZS-RE dataset named ZeroRel that supports more ZSL settings, and contribute a KG equipped with logic rules as the external knowledge of relation labels.

Statistics

Dataset# Relations (Total/Seen/Unseen)# Sentences (Total/Training/Testing)
ZeroRel100 / 70 / 30104,646 / 84,000 / 20,646

Statistically, the constructed KG contains 20,982,733 entities, 594 relations and 68,904,773 triples in total. And we contribute 50 length-1 rules and 122 length-2 rules in total for the relations in the dataset.

Usage

ZSL Dataset (Relation Splits and Original Text)

The dataset contains 3 CSV files of training samples (train.csv), seen testing samples (test_seen.csv) and unseen testing samples (test_unseen.csv), in which each row is a sample including the sentence text, the relation label, the entity mention pairs and their indexes in the sentence.

KGs and Logic Rules

4. Zero-shot Knowledge Graph Completion (ZS-KGC)

ZS-KGC here refers to predicting (completing) KG facts with relations that have never appeared in the training facts. In our resources, we employ two standard ZS-KGC benchmarks NELL-ZS and Wiki-ZS extracted from NELL and Wikidata, respectively. For each benchmark, we build an ontological schema as external knowledge, including relation semantics expressed by RDFS, such as relation and concept hierarchy, relation domain and range, relation semantics expressed by OWL, including relation characteristics (e.g., symmetry) and inter-relation relationships (e.g., composition), and textual meta data of relations and concepts.

Statistics

Dataset# Entities# Relations (Train/Val/Test)# Triples (Train/Val/Test)
NELL-ZS65,567139 / 10 / 32181,053 / 1,856 / 5,483
Wiki-ZS605,812469 / 20 / 48701,977 / 7,241 / 15,710
Ontological Schema# Relations# Concepts# Literals# Meta-relations# RDFS axioms# OWL axioms
NELL-ZS8942921,06393,055134
Wiki-ZS5601,3443,808114,821113

Usage

ZSL Datasets

Download NELL-ZS and Wiki-ZS, and put them into ZS_KGC/ZSL_Dataset/.

Each dataset contains three .json files:

   Each json file contains a dict, where keys are relations in the set, values are a list of triples of this relation.

Ontological Schemas

Each ontological schema is saved in two formats:

These files have been provided in the folder ZS_KGC/Ontological_Schema/.

4. Build KGs or Ontological Schemas Yourself

We also provided detailed construction process in ZS_IMGC/KG/data_process, ZS_RE/External_Knowledge/data_process and ZS_KGC/Ontological_Schema/data_process, you can run the scripts to build KGs or ontological schemas yourself.

Besides, we have provided temporary output files in our repository, you can also run the script output2CSV.py with different parameters to get KGs or ontological schemas with different semantic settings. For example, you can run the following command to output AwA2's KG with only class hierarchy semantics.

python output2CSV.py --dataset AwA2 --cls_hie

See more details here.

5. Related References

  1. Geng, Yuxia, Jiaoyan Chen, Xiang Zhuang, Zhuo Chen, Jeff Z. Pan, Juan Li, Zonggang Yuan, and Huajun Chen. "Benchmarking Knowledge-driven Zero-shot Learning" (Journal of Web Semantics, In Press)
  2. Geng, Yuxia, Jiaoyan Chen, Zhuo Chen, Jeff Z. Pan, Zhiquan Ye, Zonggang Yuan, Yantao Jia, and Huajun Chen. "OntoZSL: Ontology-enhanced Zero-shot Learning." In Proceedings of the Web Conference 2021, pp. 3325-3336. 2021.
  3. Chen, Jiaoyan, Yuxia Geng, Zhuo Chen, Ian Horrocks, Jeff Z. Pan, and Huajun Chen. "Knowledge-aware Zero-Shot Learning: Survey and Perspective." IJCAI Survey Track, 2021.
  4. Geng, Yuxia, Jiaoyan Chen, Zhiquan Ye, Wei Zhang, and Huajun Chen. "Explainable zero-shot learning via attentive graph convolutional network and knowledge graphs." Semantic Web Journal, vol. 12, no. 5, pp. 741-765, 2021.
  5. Chen, Jiaoyan, Freddy Lécué, Yuxia Geng, Jeff Z. Pan, and Huajun Chen. "Ontology-guided Semantic Composition for Zero-shot Learning." In Proceedings of the International Conference on Principles of Knowledge Representation and Reasoning, vol. 17, no. 1, pp. 850-854. 2020.