Home

Awesome

Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency [ACL 2024]

Environment

  1. Install required package: pip install -r requirements.txt
  2. Download benchmark dataset from google drive to data dir
  3. Download auther generated outputs from google drive [Available Soon!] to runtime dir
  4. Update api.py to your own OpenAI config

Directory Structure

|-- data # four code generation datasets
|-- runtime # runtime files including LLM generated results and inter-consistency measurements
|-- src
    |-- pipeline.py # the entry point for LLM sampling & inter-consistency measurements. All results will be saved in `runtime`.
    |-- graph.py # the entry point of MPSC
    |-- evaluation.py, _evaluation.py # evaluation metrics
    |-- execution.py, _execution.py # execution process for inter-consistency measurements
    |-- api.py # OpenAI api 
    |-- exemplars # ICL exemplars for test case generation

Reproduction

Usage of MPSC

We also provide a code snippet of MPSC for other tasks in MPSC dir.