Awesome

StructGPT: A General Framework for Large Language Model to Reason on Structured Data

This repo provides the source code & data of our paper: StructGPT: A General Framework for Large Language Model to Reason on Structured Data (Arxiv 2023).

@InProceedings{Jiang-StructGPT-2022,
  author =  {Jinhao Jiang and Kun Zhou and Zican Dong and Keming Ye and Wayne Xin Zhao and Ji-Rong Wen},
  title =   {StructGPT: A general framework for Large Language Model to Reason on Structured Data},
  year =    {2023},  
  journal={arXiv preprint arXiv:2305.09645},
  url={https://arxiv.org/pdf/2305.09645}
}

Usage

0. Requirements

You only need to install the python library and openai for querying OpenAI model API.

1. Prepare Dataset

We strongly suggest that download the processed datasets from here and then directly use them. Apart from downloading from their original website, we use the processed datasets from UnifiedSKG. After downloading our processed data, you can unzip them and put them in the /data directory.

3. Experiment

We have organized the running and evaluation scripts for each dataset under the /script directory.

3.1 Evaluation on Text-to-SQL

It is difficult to control the randomness of ChatGPT, so the reproduced results maybe a little different to the reported results.

For Spider dataset, you can directly use the following command to start running and output the evaluation results:

bash ./scripts/run_spider_wo_icl_v1.sh

Similarly, you can run the corresponding script for Spider-SYN and Spider-Realistic to get the evaluation results.

Spider_Realistic

Spider-SYN

<img src="./asset/spider_syn_wo_icl_v1.png" width="750" title="EX Result of Spider_SYN" alt=""> We save all the prediction file in *outputs/* directory.

3.2 Evaluation on TableQA

It is difficult to control the randomness of ChatGPT, so the reproduced results maybe a little different to the reported results.

For TabFact dataset, you can directly use the following command to start running and output the evaluation results:

bash ./scripts/run_tabfact_wo_icl_v1.sh

Similarly, you can run the corresponding script for WTQ and WikiSQL to get the evaluation results.

WTQ

WikiSQL

<img src="./asset/wikisql_wo_icl_v1.png" width="550" title="EX Result of WikiSQL" alt=""> We save all the prediction file in *outputs/* directory.

3.3 Evaluation on KGQA

It is difficult to control the randomness of ChatGPT, so the reproduced results maybe a little different to the reported results.

For WebQSP dataset, you can directly use the following command to start running and output the evaluation results:

bash ./scripts/run_webqsp_wo_icl_v1.sh

Similarly, you can run the corresponding script for MetaQA (1hop,2hop,3hop) to get the evaluation results.

We save all the prediction file in outputs/ directory.

Plan

Thanks for your attention. A version with better performance is on the way. Please continue to follow us!