Awesome
Contrastive Self-Supervised Learning for Commonsense Reasoning
News
- 07/17/2020: Provided source code
- 05/15/2020: Provided model for download
This repository contains the source code for our paper Contrastive Self-Supervised Learning for Commonsense Reasoning to be presented at ACL2020. The code is in parts based on the code from Huggingface Tranformers and the paper A Surprisingly Robust Trick for Winograd Schema Challenge.
We propose a self-supervised method to solve Pronoun Disambiguation and Winograd Schema Challenge problems. Our approach exploits the characteristic structure of training corpora related to so-called trigger words, which are responsible for flipping the answer in pronoun disambiguation. We achieve such commonsense reasoning by constructing pair-wise contrastive auxiliary predictions. To this end, we leverage a mutual exclusive loss regularized by a contrastive margin. Our architecture is based on the recently introduced transformer networks, BERT, that exhibits strong performance on many language understanding benchmarks. Empirical results show that our method alleviates the limitation of current supervised approaches for commonsense reasoning. This study opens up avenues for exploiting inexpensive self-supervision to achieve performance gain in commonsense reasoning tasks.
Authors:
Requirements
- Python (version 3.6 or later)
- PyTorch
- Huggingface Tranformers
Download and Installation
- Install the requiremennts:
conda install --yes --file requirements.txt
or
pip install -r requirements.txt
- Clone this repository and install dependencies:
git clone https://github.com/SAP/acl2020-commonsense-reasoning
cd acl2020-commonsense-reasoning
pip install -r requirements.txt
- Create 'data' sub-directory and download files for PDP, WSC challenge, KnowRef and DPR.:
mkdir data
wget https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/PDPChallenge2016.xml
wget https://cs.nyu.edu/faculty/davise/papers/WinogradSchemas/WSCollection.xml
wget https://raw.githubusercontent.com/aemami1/KnowRef/master/Knowref_dataset/knowref_test.json
wget http://www.hlt.utdallas.edu/~vince/data/emnlp12/train.c.txt
wget http://www.hlt.utdallas.edu/~vince/data/emnlp12/test.c.txt
cd ..
- Training and evaluating the model
python main.py --task_name wscr --do_eval --do_train --eval_batch_size 10 --data_dir "data/" --bert_model bert-large-uncased --max_seq_length 128 --train_batch_size 4 --learning_rate 1.0e-5 --alpha_param 0.05 --beta_param 0.02 --num_train_epochs 25.0 --output_dir model_output/ --gamma_param 60.0 --shuffle
- (optional) Evaluating the model without training
python main.py --task_name wscr --do_eval --eval_batch_size 10 --data_dir "data/" --bert_model bert-large-uncased --max_seq_length 128
Model
The BERT-Large model is available in the Huggingface repository as sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020.
Loading the model in Python:
tokenizer = AutoTokenizer.from_pretrained("sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020")
model = AutoModelWithLMHead.from_pretrained("sap-ai-research/BERT-Large-Contrastive-Self-Supervised-ACL2020")
This model should reproduce the results reported in the paper:
Knowref-test: 0.6558966074313409
DPR/WSCR-test: 0.8014184397163121
WSC: 0.6959706959706959
PDP: 0.9
Related work
See our work accepted ACL'19 - Attention Is (not) All You Need for Commonsense Reasoning - proposing BERT attention-guidance for commonsense reasoning. arXiv, GitHub
Known Issues
No issues known
How to obtain support
This project is provided "as-is" and any bug reports are not guaranteed to be fixed.
Citations
If you use this code in your research, please cite:
@inproceedings{klein-nabi-2020-contrastive,
title = "Contrastive Self-Supervised Learning for Commonsense Reasoning",
author = "Klein, Tassilo and
Nabi, Moin",
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
month = jul,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.acl-main.671",
pages = "7517--7523"
}
License
Copyright (c) 2024 SAP SE or an SAP affiliate company. All rights reserved. This project is licensed under the Apache Software License, version 2.0 except as noted otherwise in the LICENSE.