Awesome

DevBench: Towards LLMs based Automated Software Development

👋 <a href="#-overview">Overview</a> | 📖 <a href="#-benchmarking-code-llms">Benchmarking</a> | ⚙️ <a href="#-set-up-with-docker">Setup</a> | 🚀 <a href="#-usage">Usage</a> | 🔎 <a href="#-citation">Citation</a> | 📄 <a href="#-license">License</a>

📬 Contact: libowen.ne@gmail.com, chao.peng@acm.org

📝 Check out our paper HERE !

👋 Overview

• DevBench is a comprehensive benchmark designed to evaluate LLMs across various stages of the software development lifecycle, including software design, environment setup, implementation, acceptance testing, and unit testing. By integrating these interconnected steps under a single framework, DevBench offers a holistic perspective on the potential of LLMs for automated software development.

• The DevBench dataset comprises 22 curated repositories across 4 programming languages (Python, C/C++, Java, JavaScript), covering a wide range of domains such as machine learning, databases, web services, and command-line utilities.

  <p align="center"> <img src='./assets/preview.png' width=860> </p>

• DevBench includes a comprehensive and automatic evaluation suite for all tasks involved. We provide extensive acceptance and unit test cases for the implementation task 🤗. Additionally, we utilize LLM-as-a-Judge for evaluating the software design task 👩🏽‍⚖️. Further details on our task specifications can be found here.

  <p align="center"> <img src='./assets/tasks.png' width=860> </p>

• We have developed a baseline agent system based on the popular multi-agent software development system, ChatDev. Special thanks to our collaborators at ChatDev!

📖 Benchmarking Code LLMs

Evaluation results of the coding tasks on DevBench.

Evaluation results of the software design on DevBench.

The code for the software design evaluation can be found here👩🏽‍⚖️.

🐳 Set Up with Docker

For a secure and isolated environment, we offer Docker support for DevBench. Please refer to our detailed Installation Guide.

🚀 Usage

1. Prepare the environment variables

Add your DevBench directory to your PYTHONPATH variable.

export PYTHONPATH="${PYTHONPATH}:${path_to_devbench}"

For running the benchmark_data/java/Actor_relationship_game repo, configure your TMDB key.

export TMDB_API_KEY=${your_TMDB_key}

2. Prepare the chat models

OpenAI GPT models

Set your OpenAI API key as an environment variable.

export OPENAI_API_KEY="your_OpenAI_API_key"

Open source models

For deploying open source models, please refer to lmdeploy or vllm.

After the deployment, please configure the IP address in open_source_model.json.

For codellama and deepseek-coder models, which are integrated into our experiments, simply fill in the IP address in {"model_name": $model_ip_address}.

For example：

{
  "codellama-7b-instruct": "",
  "codellama-13b-instruct": "",
  "codellama-34b-instruct": "",
  "deepseek-coder-1.3b-instruct": "",
  "deepseek-coder-6.7b-instruct": "",
  "deepseek-coder-33b-instruct": "$model_ip_address"
}

For additional models, add a new field as shown below.

{
  "customized-model": {"$model_name": "$model_ip_address"}
}

3. Run the agent system

Run script

cd agent_sysyem/baseline
python run.py --config Implementation --input_path ../../benchmark_data/python/TextCNN/ --model gpt-4-turbo-new --model_source openai  --review execution --evaluate

Parameters

• config (str) - Specifies the task in the DevBench: SoftwareDesign | EnvironmentSetup | Implementation | AcceptanceTesting | UnitTesting.
• input_path (str) - Specifies the repo path.
• project_name (str) - Specifies the repo name. If empty, defaults to the last segment of input_path (i.e., input_path.split('/')[-1])
• model (str) - Specifies the name of the language model: gpt-3.5-turbo | gpt-4 | gpt-4-32k | gpt-4-turbo | claude-2 | claude-2.1 | codellama-7b | codellama-13b | codellama-34b | deepseek-coder-1.3b | deepseek-coder-6.7b | deepseek-coder-33b | customized-model.
• customized_model_name (Optional, str) - Specifies the custom model name if the value of the model parameter is customized-model.
• model_source (str) - Specifies the model type, open source model or openai closed source model : open_source ｜ openai
• review (str) - Specifies the review mode: none | normal | execution.
  • none: a single forward pass of Coding.
  • normal: Coding and CodeReview in alternation, with CodeReview lacking program execution feedback.
  • execution: Coding and CodeReview in alternation, with CodeReview including program execution feedback.
• read_src_code (bool) - Whether to read source code in the AcceptanceTesting and UnitTesting tasks.
• evaluate (bool) - Whether to evaluate in the end. The evaluation for the software design can be found here.
• temperature (float) - temperature
• top_p (float) - top_p

When you use normal review and execution review, the cyclenum parameter of CompanyConfig/{task_name}/ChatChainConfig.json can be specified as the number of rounds of review. The default is 2.

🔎 Citation

@article{li2024devbench,
  title={DevBench: A Comprehensive Benchmark for Software Development},
  author={Li, Bowen and Wu, Wenhan and Tang, Ziwei and Shi, Lin and Yang, John and Li, Jinyang and Yao, Shunyu and Qian, Chen and Hui, Binyuan and Zhang, Qicheng and others},
  journal={arXiv preprint arXiv:2403.08604},
  year={2024}
}

📄 License

• Source Code Licensing: Our project's source code is licensed under the Apache 2.0 License. This license permits the use, modification, and distribution of the code, subject to certain conditions outlined in the Apache 2.0 License.
• Data Licensing: The related data utilized in our project is licensed under CC BY 4.0, which allows anyone to copy, distribute, transmit, adapt and make commercial use of the dataset.