Home

Awesome

<div align="center"> <img src="docs/en/_static/image/logo.svg" width="500px"/> <br /> <br />

<br>

<!-- [![PyPI](https://badge.fury.io/py/opencompass.svg)](https://pypi.org/project/opencompass/) -->

🌐Website | 📖CompassHub | 📊CompassRank | 📘Documentation | 🛠️Installation | 🤔Reporting Issues

English | 简体中文

</div> <p align="center"> 👋 join us on <a href="https://discord.gg/KKwfEbFj7U" target="_blank">Discord</a> and <a href="https://r.vansin.top/?r=opencompass" target="_blank">WeChat</a> </p>

[!IMPORTANT]

Star Us, You will receive all release notifications from GitHub without any delay ~ ⭐️

📣 OpenCompass 2.0

We are thrilled to introduce OpenCompass 2.0, an advanced suite featuring three key components: CompassKit, CompassHub, and CompassRank. oc20

CompassRank has been significantly enhanced into the leaderboards that now incorporates both open-source benchmarks and proprietary benchmarks. This upgrade allows for a more comprehensive evaluation of models across the industry.

CompassHub presents a pioneering benchmark browser interface, designed to simplify and expedite the exploration and utilization of an extensive array of benchmarks for researchers and practitioners alike. To enhance the visibility of your own benchmark within the community, we warmly invite you to contribute it to CompassHub. You may initiate the submission process by clicking here.

CompassKit is a powerful collection of evaluation toolkits specifically tailored for Large Language Models and Large Vision-language Models. It provides an extensive set of tools to assess and measure the performance of these complex models effectively. Welcome to try our toolkits for in your research and products.

<details> <summary><kbd>Star History</kbd></summary> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/svg?repos=open-compass%2Fopencompass&theme=dark&type=Date"> <img width="100%" src="https://api.star-history.com/svg?repos=open-compass%2Fopencompass&type=Date"> </picture> </details>

🧭 Welcome

to OpenCompass!

Just like a compass guides us on our journey, OpenCompass will guide you through the complex landscape of evaluating large language models. With its powerful algorithms and intuitive interface, OpenCompass makes it easy to assess the quality and effectiveness of your NLP models.

🚩🚩🚩 Explore opportunities at OpenCompass! We're currently hiring full-time researchers/engineers and interns. If you're passionate about LLM and OpenCompass, don't hesitate to reach out to us via email. We'd love to hear from you!

🔥🔥🔥 We are delighted to announce that the OpenCompass has been recommended by the Meta AI, click Get Started of Llama for more information.

Attention<br /> We launch the OpenCompass Collaboration project, welcome to support diverse evaluation benchmarks into OpenCompass! Clike Issue for more information. Let's work together to build a more powerful OpenCompass toolkit!

🚀 What's New <a><img width="35" height="20" src="https://user-images.githubusercontent.com/12782558/212848161-5e783dd6-11e8-4fe0-bbba-39ffb77730be.png"></a>

More

✨ Introduction

image

OpenCompass is a one-stop platform for large model evaluation, aiming to provide a fair, open, and reproducible benchmark for large model evaluation. Its main features include:

📊 Leaderboard

We provide OpenCompass Leaderboard for the community to rank all public models and API models. If you would like to join the evaluation, please provide the model repository URL or a standard API interface to the email address opencompass@pjlab.org.cn.

<p align="right"><a href="#top">🔝Back to top</a></p>

🛠️ Installation

Below are the steps for quick installation and datasets preparation.

💻 Environment Setup

Open-source Models with GPU

conda create --name opencompass python=3.10 pytorch torchvision pytorch-cuda -c nvidia -c pytorch -y
conda activate opencompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .

API Models with CPU-only

conda create -n opencompass python=3.10 pytorch torchvision torchaudio cpuonly -c pytorch -y
conda activate opencompass
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
# also please install requirements packages via `pip install -r requirements/api.txt` for API models if needed.

📂 Data Preparation

You can choose one for the following method to prepare datasets.

Offline Preparation

You can download and extract the datasets with the following commands:

# Download dataset to data/ folder
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip

Automatic Download from OpenCompass

We have supported download datasets automatic from the OpenCompass storage server. You can run the evaluation with extra --dry-run to download these datasets. Currently, the supported datasets are listed in here. More datasets will be uploaded recently.

(Optional) Automatic Download with ModelScope

Also you can use the ModelScope to load the datasets on demand.

Installation:

pip install modelscope[framework]
export DATASET_SOURCE=ModelScope

Then submit the evaluation task without downloading all the data to your local disk. Available datasets include:

humaneval, triviaqa, commonsenseqa, tydiqa, strategyqa, cmmlu, lambada, piqa, ceval, math, LCSTS, Xsum, winogrande, openbookqa, AGIEval, gsm8k, nq, race, siqa, mbpp, mmlu, hellaswag, ARC, BBH, xstory_cloze, summedits, GAOKAO-BENCH, OCNLI, cmnli

Some third-party features, like Humaneval and Llama, may require additional steps to work properly, for detailed steps please refer to the Installation Guide.

<p align="right"><a href="#top">🔝Back to top</a></p>

🏗️ ️Evaluation

After ensuring that OpenCompass is installed correctly according to the above steps and the datasets are prepared, you can evaluate the performance of the LLaMA-7b model on the MMLU and C-Eval datasets using the following command:

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl

Additionally, if you want to use an inference backend other than HuggingFace for accelerated evaluation, such as LMDeploy or vLLM, you can do so with the command below. Please ensure that you have installed the necessary packages for the chosen backend and that your model supports accelerated inference with it. For more information, see the documentation on inference acceleration backends here. Below is an example using LMDeploy:

python run.py --models hf_llama_7b --datasets mmlu_ppl ceval_ppl -a lmdeploy

OpenCompass has predefined configurations for many models and datasets. You can list all available model and dataset configurations using the tools.

# List all configurations
python tools/list_configs.py
# List all configurations related to llama and mmlu
python tools/list_configs.py llama mmlu

You can also evaluate other HuggingFace models via command line. Taking LLaMA-7b as an example:

python run.py --datasets ceval_ppl mmlu_ppl --hf-type base --hf-path huggyllama/llama-7b

[!TIP]

configuration with _ppl is designed for base model typically. configuration with _gen can be used for both base model and chat model.

Through the command line or configuration files, OpenCompass also supports evaluating APIs or custom models, as well as more diversified evaluation strategies. Please read the Quick Start to learn how to run an evaluation task.

<p align="right"><a href="#top">🔝Back to top</a></p>

📖 Dataset Support

<table align="center"> <tbody> <tr align="center" valign="bottom"> <td> <b>Language</b> </td> <td> <b>Knowledge</b> </td> <td> <b>Reasoning</b> </td> <td> <b>Examination</b> </td> </tr> <tr valign="top"> <td> <details open> <summary><b>Word Definition</b></summary> </details> <details open> <summary><b>Idiom Learning</b></summary> </details> <details open> <summary><b>Semantic Similarity</b></summary> </details> <details open> <summary><b>Coreference Resolution</b></summary> </details> <details open> <summary><b>Translation</b></summary> </details> <details open> <summary><b>Multi-language Question Answering</b></summary> </details> <details open> <summary><b>Multi-language Summary</b></summary> </details> </td> <td> <details open> <summary><b>Knowledge Question Answering</b></summary> </details> </td> <td> <details open> <summary><b>Textual Entailment</b></summary> </details> <details open> <summary><b>Commonsense Reasoning</b></summary> </details> <details open> <summary><b>Mathematical Reasoning</b></summary> </details> <details open> <summary><b>Theorem Application</b></summary> </details> <details open> <summary><b>Comprehensive Reasoning</b></summary> </details> </td> <td> <details open> <summary><b>Junior High, High School, University, Professional Examinations</b></summary> </details> <details open> <summary><b>Medical Examinations</b></summary> </details> </td> </tr> </td> </tr> </tbody> <tbody> <tr align="center" valign="bottom"> <td> <b>Understanding</b> </td> <td> <b>Long Context</b> </td> <td> <b>Safety</b> </td> <td> <b>Code</b> </td> </tr> <tr valign="top"> <td> <details open> <summary><b>Reading Comprehension</b></summary> </details> <details open> <summary><b>Content Summary</b></summary> </details> <details open> <summary><b>Content Analysis</b></summary> </details> </td> <td> <details open> <summary><b>Long Context Understanding</b></summary> </details> </td> <td> <details open> <summary><b>Safety</b></summary> </details> <details open> <summary><b>Robustness</b></summary> </details> </td> <td> <details open> <summary><b>Code</b></summary> </details> </td> </tr> </td> </tr> </tbody> </table>

📖 Model Support

<table align="center"> <tbody> <tr align="center" valign="bottom"> <td> <b>Open-source Models</b> </td> <td> <b>API Models</b> </td> <!-- <td> <b>Custom Models</b> </td> --> </tr> <tr valign="top"> <td> </td> <td> </td> </tr> </tbody> </table> <p align="right"><a href="#top">🔝Back to top</a></p>

🔜 Roadmap

👷‍♂️ Contributing

We appreciate all contributions to improving OpenCompass. Please refer to the contributing guideline for the best practice.

<!-- Copy-paste in your Readme.md file --> <!-- Made with [OSS Insight](https://ossinsight.io/) --> <a href="https://github.com/open-compass/opencompass/graphs/contributors" target="_blank"> <table> <tr> <th colspan="2"> <br><img src="https://contrib.rocks/image?repo=open-compass/opencompass"><br><br> </th> </tr> </table> </a>

🤝 Acknowledgements

Some code in this project is cited and modified from OpenICL.

Some datasets and prompt implementations are modified from chain-of-thought-hub and instruct-eval.

🖊️ Citation

@misc{2023opencompass,
    title={OpenCompass: A Universal Evaluation Platform for Foundation Models},
    author={OpenCompass Contributors},
    howpublished = {\url{https://github.com/open-compass/opencompass}},
    year={2023}
}
<p align="right"><a href="#top">🔝Back to top</a></p>