Home

Awesome

Instruction Induction

This repository contains the code and data of the paper: "Instruction Induction: From Few Examples to Natural Language Task Descriptions"

Data

The data for the instruction induction experiments, as well as for the execution accuracy evaluation, are available in the data folder.

Usage

Setup

Install the required packages using pip install -r requirements.txt.

Instructions Prediction

To run the instruction induction experiments, run the following command:

python induction.py \
--engine $OPENAI_ENGINE \
--organization $OPENAI_ORGANIZATION \
--api_key $OPENAI_API_KEY \
--data_dir $INPUT_DATA_DIR \
--out_dir $OUTPUT_DIR \
--max_tokens $MAX_TOKENS \
--tasks $TASK_LIST

where

Postprocessing

We apply a postprocessing protocol, which includes a basic cleanup for the generated instructions as well as grouping identical instructions, to speedup and reduce the cost of the execution accuracy experiments. To postprocess the generated instructions, run

python postprocess_instructions.py \
--engine $OPENAI_ENGINE \
--predictions_dir $PREDICTIONS_DIR \
--tasks $TASK_LIST

where

Evaluation

To measure the execution accuracy of the generated instructions, first run the following command:

python prepare_for_execution.py \
--model_name $OPENAI_ENGINE \
--execute_data_dir $EXECUTE_DATA_DIR \
--predictions_dir $PREDICTIONS_DIR \
--out_dir $OUTPUT_DIR \
--tasks $TASK_LIST

where

Next, to execute the instructions, run

python execute_instructions.py \
--execution_engine $OPENAI_EXECUTION_ENGINE \
--instruction_generation_model $INSTRUCTION_GENERATION_MODEL \
--organization $OPENAI_ORGANIZATION \
--api_key $OPENAI_API_KEY \
--input_dir $INPUT_DATA_DIR \
--out_dir $OUTPUT_DIR \
--max_tokens $MAX_TOKENS \
--tasks $TASK_LIST

where

Finally, to obtain the execution accuracy scores, run the following command:

python evaluate.py \
--instruction_generation_model $INSTRUCTION_GENERATION_MODEL \
--execution_input_dir $INPUT_DATA_DIR \
--predictions_dir $PREDICTIONS_DIR \
--tasks $TASK_LIST

where

Citation

@misc{honovich2022induction,
      title={Instruction Induction: From Few Examples to Natural Language Task Descriptions},
      author={Honovich, Or and Shaham, Uri and Bowman, Samuel R. and Levy, Omer},
      year={2022},
      eprint={2205.10782},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}