Awesome
Generated Knowledge Prompting
This repository contains the code for our ACL 2022 paper, Generated Knowledge Prompting for Commonsense Reasoning.
Installation
conda env create -f environment.yml
conda activate gkp
Download and Standardize the data
First, download the dataset for the following tasks: NumerSense, CommonsenseQA, CommonsenseQA 2.0, and QASC.
Next, use standardize/{task_name}_standardize.py
to put the data in a unified format.
For convenience, we provide the standardized data at data/{task_name}/{split_name}.{task_name}.json
Generate Knowledge
Use gpt3_generate_knowledge.py
to generate knowledge for a task dataset.
For example, to generate knowledge for the validation set of NumerSense, run
python knowledge/gpt3_generate_knowledge.py \
--task numersense \
--input_path data/numersense/validation.json \
--output_path data/numersense/knowledge/knowledge_gpt3.validation.json \
--prompt_path knowledge/prompts/numersense_prompt.txt
This will create a JSON file that contains the knowledge statements for each question in this dataset.
This step needs access to GPT-3.
For convenience, we provide the knowledge files that were produced by our method and were used in downstream evaluation.
These files are at data/{task_name}/knowledge/knowledge_gpt3.{split_name}.{task_name}.json
Inference with Knowledge Prompting
For the NumerSense task, use infer_numersense_t5.py
to run inference with the generated knowledge.
For example:
CUDA_VISIBLE_DEVICES=0 python inference/infer_numersense_t5.py \
--model-type t5-11b \
--input-path data/numersense/knowledge/knowledge_gpt3.validation.json
where --input-path
is the knowledge file we produced in the previous step at the --output-path
.
This will create a JSON file that contains the inference results under data/{task_name}/inference/
.
For the CommonsenseQA, CommonsenseQA 2.0, and QASC tasks, use infer_t5.py
instead.
For example:
CUDA_VISIBLE_DEVICES=0 python inference/infer_t5.py \
--task csqa \
--model-type allenai/unifiedqa-t5-11b \
--input-path data/csqa/knowledge/knowledge_gpt3.dev.csqa.json
To get the baseline of your inference model (i.e. without using generated knowledge), use the standardized data file as --input-path
:
CUDA_VISIBLE_DEVICES=0 python inference/infer_t5.py \
--task csqa \
--model-type allenai/unifiedqa-t5-11b \
--input-path data/csqa/dev.csqa.json
Please check the argparse choices in the scripts for more parameters available.
A special note on the QASC dataset
On the QASC dataset, we find that the text-infilling question format is more suitable when using the off-the-shelf T5 as inference model. Therefore, we first convert the dataset to an infilling format:
python standardize/qasc_to_infill.py
For convenience, we include the infilling-format QASC dataset at data/qasc/{split_name}.qasc_infill.json
.
For the knowledge generation step, still use the original standardized data file. For the inference step, use the knowledge file where questions are in the text-infilling format:
CUDA_VISIBLE_DEVICES=0 python inference/infer_t5.py \
--task qasc \
--model-type t5-11b \
--input-path data/qasc/knowledge/knowledge_gpt3.dev.qasc_infill.json
Note that it is not needed to use the infilling-format data with finetuned T5 or any UnifiedQA inference models.
Citation
If you find this repository helpful, please consider citing our paper:
@article{liu2021generated,
title={Generated knowledge prompting for commonsense reasoning},
author={Liu, Jiacheng and Liu, Alisa and Lu, Ximing and Welleck, Sean and West, Peter and Bras, Ronan Le and Choi, Yejin and Hajishirzi, Hannaneh},
journal={arXiv preprint arXiv:2110.08387},
year={2021}
}