Awesome

ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models

This repo provides the source code & data of our paper: ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models (Arxiv 2023).

@InProceedings{Chen-ChatCot-2023,
      title = {ChatCoT: Tool-Augmented Chain-of-Thought Reasoning on Chat-based Large Language Models}, 
      author = {Zhipeng Chen and Kun Zhou and Beichen Zhang and Zheng Gong and Wayne Xin Zhao and Ji-Rong Wen},
      year = {2023},
      eprint = {2305.14323},
      archivePrefix = {arXiv},
      primaryClass = {cs.CL}
}

Data & Code

• math/: the code about ChatCoT on MATH dataset
  • demo/: the demos used in in-context learning on few-shot setting
  • math/result/: the result files of different methods
  • math/scripts/: the running scripts of different methods
  • math/ablation: the code of ablation study
  • math/self_consistency: the code to explore combining ChatCoT with CoT improvement strategies

The hotpotqa/ folder is similar with math/ folder

Usage

Prepare

You can use following scripts to install related python package through pip:

git clone https://github.com/RUCAIBox/ChatCoT.git
cd ChatCoT
pip install -r requirements.txt

Inference

You can run ChatCot on the sub-task of MATH dataset by running run_turbo_chatcot.sh:

cd math
bash scripts/run_turbo_chatcot.sh

You have to replace YOUR_API_KEY with you openai api key in the code. Specially, we run ChatCoT through multi-processing, and you should prepare a list of api key in order to run the code correctly.

Evaluate

You can evaluate the results by running eval.sh:

cd math
bash scripts/eval.sh

Results

Main Results

Ablation Study

The results of ablation study. TK, RATK, and MRF denote if using tool knowledge, retrieval-augmented task knowledge, and multi-turn reasoning format at early turns of the conversation, respectively.

Combining CoT Improvement Strategies