Awesome
Grade-School Math with Irrelevant Context (GSM-IC)
This repository contains the dataset Grade-School Math with Irrelevant Context (GSM-IC) used in this paper: Large Language Models Can Be Easily Distracted by Irrelevant Context.
Data Format
GSM8K_validation.jsonl
: the development split of GSM8K dataset used in the experiments.
Field name | Value |
---|---|
question | Input question. |
answer | The ground truth answer. |
n_steps | The number of intermediate steps to calculate the answer. |
GSM-IC_2step.json
: GSM-IC split with problems that require 2 intermediate steps.
Field name | Value |
---|---|
original_question | Original question from the GSM8K development set. |
new_question | The new question with irrelevant context added to the original question. |
answer | The ground truth answer. |
n_steps | The number of intermediate steps to calculate the answer. |
role_label, number_label, sentence_label | Categories of the added irrelevant context. Needed for result analysis, not needed for model prediction. |
role, number, sentence_template | Added irrelevant context. Not needed for experiments. |
GSM-IC_mstep.json
: GSM-IC split with problems that require more than 2 intermediate steps. Same format asGSM-IC_2step.json
.
Citation
If you use the data released through this repository, please cite the following paper:
@article{shi2023large,
title={Large Language Models Can Be Easily Distracted by Irrelevant Context},
author={Shi, Freda and Chen, Xinyun and Misra, Kanishka and Scales, Nathan and Dohan, David and Chi, Ed and Schärli, Nathanael and Zhou, Denny},
journal={arXiv preprint arXiv:2302.00093},
year={2023}
}