Home

Awesome

Program of Thoughts

This is code repository for the TMLR2023 paper Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks.

We propose to disentangle computation and reasoning from in the problem solving process. The large language model only needs to express the thoughts using Python program, the computation and solving process is accomplished via an external Python Interpreter.

<p align="center"> <img src="assets/intro.001.jpeg" width="1000"> </p>
  1. We outperform few-shot CoT by an average of 12% on all the datasets evaluated.
  2. We outperform zero-shot CoT also by an average of 12% on all the datasets evaluated.
  3. We achieve SoTA performance with self-consistency decoding on all the evaluated math word problem datasets (GSM8K, AQuA, SVAMP, TabMWP, MultiArith).

Comparison with Few-shot CoT:

<p align="center"> <img src="assets/FS_comparison.png" width="600"> </p>

Comparison with Few-shot CoT with self-consistency:

<p align="center"> <img src="assets/SC_comparison.png" width="600"> </p>

Comparison with Zero-shot CoT:

<p align="center"> <img src="assets/ZS_comparison.png" width="600"> </p>

News

  1. Added CoT evaluation for AQuA QA for GPT4, the accuracy is 72.7%
  2. Adding Benchmark

Running the code

First you need to specify your OPENAI key

export OPENAI_KEY = [YOUR_KEY]
python run_gsm8k.py --greedy
python run_aqua.py --greedy
...
python run_gsm8k.py
python run_aqua.py
...
python run_gsm8k_zs.py
python run_aqua_zs.py
...

The prediction file will be dumped in the outputs/ folder, let's say gsm8K_s0_e-1_11_17_10_20.jsonl, or gsm8K_sc_s0_e-1_11_08_21_14.jsonl, or gsm8K_zs_s0_e-1_11_19_09_55.jsonl.

cd outputs
python compute_score.py --inputs gsm8K_s0_e-1_11_17_10_20.jsonl
python compute_score.py --inputs aqua_s0_e-1_11_06_18_38.jsonl
python compute_score.py --inputs svamp_s0_e-1_11_06_21_11.jsonl
....

Few-shot Results

  1. GSM8K
  1. AQuA
  1. SVAMP
  1. TabWMP
  1. FinQA
  1. ConvFinQA
  1. TATQA

Zero-shot Results

  1. GSM8K
  1. AQuA
python compute_score.py --inputs aqua_zs_s0_e-1_11_19_11_56.jsonl --relaxed
  1. SVAMP
  1. MultiArith
  1. TabMWP

Cite our Work

@article{chen2022program,
  title = {Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks},
  author = {Wenhu Chen and Xueguang Ma and Xinyi Wang and William W. Cohen},
  journal={Transactions on Machine Learning Research},
  year = {2023},
}