Home

Awesome

Prompt-lib

TLDR: This library makes it easy to write prompts for few-shot prompting tasks. It includes rate-limiting and retry logic, and makes it easy to write prompts for different tasks.

export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
python prompt_lib/run_inference.py --task_id boolsimplify_stream --num_prompt_examples -1 --name boolsimplify_stream_code-davinci-002_s1 --model_name code-davinci-002 --max_tokens 600 --seed 1 --num_questions_per_thread 500 --temperature 0.0 --num_inference_examples 3 --cot_task --is_debug

News 📢

[Aug 2023] Added support for Together.ai API.

[June 2023] Added support for claude series of models from Anthropic.

[Mar 2023] prompt-lib now works with ChatGPT/GPT-4. The addition is backward compatible, so you can simply usegpt-3.5-turbo or gpt-4 as the engine name. However, you may want to checkout how we convert the prompt to messages here.

Colabs

You can run these colabs directly in the browser. Supporting bulk-inference is the main goal of prompt-lib, but these notebooks should give you a good idea of some of the functionality.

If your goal is not to do bulk inference (typically required for research papers/benchmarking), we recommend checking out https://langchain.readthedocs.io/.

Running on a custom task

prompt-lib comes with a large number of tasks. To register a new task, please follow these instructions:

Step 1: Defining a custom task

@dataclass
class Example:
    question: str
    answer: str
    thought: str
[
    Example(
        question="!a & a & (a ^ c & d | !b)",
        answer="false",
        thought="false & expr is false",
    ),
    Example(
        question="a & b | a & c",
        answer="a & (b | c)",
        thought="a & (b | c) is the same as a & b | a & c",
    ),
    Example(
        question="a & b | a & b",
        answer="a & b",
        thought="expr | expr is the same as expr",
    ),
    Example(
        question="(a | !a) | ((b & !c | c & !b) & (a | !a) | (b & !c | c & !b))",
        answer="true",
        thought="true | expr is true",
    ),
]
 bool_simple_taskid_to_prompt = {
    "boolsimplify_stream": bool_simple_examples
}

Text-file based prompt:

 bool_simple_taskid_to_prompt = {
    "boolsimplify_txt": "prompts/boolsimplify/boolsimplify.txt"
}

Step 2: Registering the task

from prompt_lib.prompts.boolsimplify import boolsimplify_taskid_to_prompt

task_id_to_prompt.update(bool_simple_taskid_to_prompt)

Step 3: Defining inputs/outputs

{
    "input": "a & b | a & c",
    "target": "a & (b | c)",
}

Step 4: Running the task

export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
bash scripts/run_single_task.sh boolsimplify_stream

Structure of a prompt

Additional notes

python prompt-lib/scripts/shuffle_prompt.py --prompt_path quco_prompts/gsm/function_with_comments.txt --seeds 1 2 3 --example_sep $'\n\n\n'

(note the $'\n\n\n' to separate examples--this is needed for passing newlines to the script)

Custom Evaluation

Multiple outputs

For example:

python3 prompt_lib/run_inference.py --task_id boolsimplify_stream --num_prompt_examples -1 --name boolsimplify_stream_code-davinci-002_s1 --model_name code-davinci-002 --max_tokens 600 --seed 1 --num_questions_per_thread 500 --temperature 0.9 --num_inference_examples 3 --cot_task --is_debug --num_completions 3

Generates an output file data/logs/boolsimplify_stream/code-davinci-002/temp_0.9/seed_1/k_all/2023-01-02_10-16-39/outputs.jsonl (the timestamp will be different). One entry in the file is:

{
    "prompt": "Q: a & b | a & c\nA: a & (b | c) is the same as a & b | a & c The answer is a & (b | c)\n\n\nQ: a & b | a & b\nA: expr | expr is the same as expr The answer is a & b\n\n\nQ: !a & a & (a ^ c & d | !b)\nA: false & expr is false The answer is false\n\n\nQ: (a | !a) | ((b & !c | c & !b) & (a | !a) | (b & !c | c & !b))\nA: true | expr is true The answer is true",
    "question": "\nQ: a & b | a & b\nA: ",
    "answer": "a & b",
    "generated_answers": [
        "expr | expr is the same as expr The answer is a & b",
        "expr | expr is the same as expr The answer is a & b",
        "!b & a is the same as a & !b The answer is !b & a"
    ],
    "generated_answer": "expr | expr is the same as expr The answer is a & b",
    "is_correct": 1
}

TODOs: