Awesome

Multi-SWE-bench Website

This repository contains the code for the website and leaderboard of the Multi-SWE-bench project.

To learn more about Multi-SWE-bench, please check out the main code repository along with the main paper, SWE-bench-java: A GitHub Issue Resolving Benchmark for Java.

📄 Citation

If you found SWE-bench or Multi-SWE-bench helpful for your work, please cite as follows:

@inproceedings{jimenez2024swebench,
    title={{SWE}-bench: Can Language Models Resolve Real-world Github Issues?},
    author={Carlos E Jimenez and John Yang and Alexander Wettig and Shunyu Yao and Kexin Pei and Ofir Press and Karthik R Narasimhan},
    booktitle={The Twelfth International Conference on Learning Representations},
    year={2024},
    url={https://openreview.net/forum?id=VTF8yNQM66}
}

@misc{zan2024swebenchjava,
      title={SWE-bench-java: A GitHub Issue Resolving Benchmark for Java}, 
      author={Daoguang Zan and Zhirong Huang and Ailun Yu and Shaoxin Lin and Yifan Shi and Wei Liu and Dong Chen and Zongshuai Qi and Hao Yu and Lei Yu and Dezhi Ran and Muhan Zeng and Bo Shen and Pan Bian and Guangtai Liang and Bei Guan and Pengjie Huang and Tao Xie and Yongji Wang and Qianxiang Wang},
      year={2024},
      eprint={2408.14354},
      archivePrefix={arXiv},
      primaryClass={cs.SE},
      url={https://arxiv.org/abs/2408.14354}, 
}

🙏 Acknowledgements

We express our deepest gratitude to the creators of the SWE-bench dataset, whose foundational work our project is built upon. Additionally, we would like to thank the creators of the SQuAD dataset, whose template is used for SWE-bench.