Home

Awesome

<p align="center"> English | <a href="README_zh.md">็ฎ€ไฝ“ไธญๆ–‡</a> </p> <p align="center"> <a href="https://badge.fury.io/py/evalscope"><img src="https://badge.fury.io/py/evalscope.svg" alt="PyPI version" height="18"></a> <a href="https://pypi.org/project/evalscope"><img alt="PyPI - Downloads" src="https://static.pepy.tech/badge/evalscope"> </a> <a href='https://evalscope.readthedocs.io/en/latest/?badge=latest'> <img src='https://readthedocs.org/projects/evalscope-en/badge/?version=latest' alt='Documentation Status' /> </a> <br> <a href="https://evalscope.readthedocs.io/en/latest/">๐Ÿ“– Documents</a> <p>

๐Ÿ“‹ Table of Contents

๐Ÿ“ Introduction

EvalScope is the official model evaluation and performance benchmarking framework launched by the ModelScope community. It comes with built-in common benchmarks and evaluation metrics, such as MMLU, CMMLU, C-Eval, GSM8K, ARC, HellaSwag, TruthfulQA, MATH, and HumanEval. EvalScope supports various types of model evaluations, including LLMs, multimodal LLMs, embedding models, and reranker models. It is also applicable to multiple evaluation scenarios, such as end-to-end RAG evaluation, arena mode, and model inference performance stress testing. Moreover, with the seamless integration of the ms-swift training framework, evaluations can be initiated with a single click, providing full end-to-end support from model training to evaluation ๐Ÿš€

<p align="center"> <img src="docs/en/_static/images/evalscope_framework.png" width="70%"> <br>EvalScope Framework. </p>

The architecture includes the following modules:

  1. Model Adapter: The model adapter is used to convert the outputs of specific models into the format required by the framework, supporting both API call models and locally run models.
  2. Data Adapter: The data adapter is responsible for converting and processing input data to meet various evaluation needs and formats.
  3. Evaluation Backend:
    • Native: EvalScopeโ€™s own default evaluation framework, supporting various evaluation modes, including single model evaluation, arena mode, baseline model comparison mode, etc.
    • OpenCompass: Supports OpenCompass as the evaluation backend, providing advanced encapsulation and task simplification, allowing you to submit tasks for evaluation more easily.
    • VLMEvalKit: Supports VLMEvalKit as the evaluation backend, enabling easy initiation of multi-modal evaluation tasks, supporting various multi-modal models and datasets.
    • RAGEval: Supports RAG evaluation, supporting independent evaluation of embedding models and rerankers using MTEB/CMTEB, as well as end-to-end evaluation using RAGAS.
    • ThirdParty: Other third-party evaluation tasks, such as ToolBench.
  4. Performance Evaluator: Model performance evaluation, responsible for measuring model inference service performance, including performance testing, stress testing, performance report generation, and visualization.
  5. Evaluation Report: The final generated evaluation report summarizes the model's performance, which can be used for decision-making and further model optimization.
  6. Visualization: Visualization results help users intuitively understand evaluation results, facilitating analysis and comparison of different model performances.

๐ŸŽ‰ News

๐Ÿ› ๏ธ Installation

Method 1: Install Using pip

We recommend using conda to manage your environment and installing dependencies with pip:

  1. Create a conda environment (optional)

    # It is recommended to use Python 3.10
conda create -n evalscope python=3.10
# Activate the conda environment
conda activate evalscope

  2. Install dependencies using pip

    pip install evalscope                # Install Native backend (default)
# Additional options
pip install evalscope[opencompass]   # Install OpenCompass backend
pip install evalscope[vlmeval]       # Install VLMEvalKit backend
pip install evalscope[all]           # Install all backends (Native, OpenCompass, VLMEvalKit)

[!WARNING] As the project has been renamed to evalscope, for versions v0.4.3 or earlier, you can install using the following command:

pip install llmuses<=0.4.3

To import relevant dependencies using llmuses:

from llmuses import ...

Method 2: Install from Source

  1. Download the source code

    git clone https://github.com/modelscope/evalscope.git

  2. Install dependencies

    cd evalscope/
pip install -e .                  # Install Native backend
# Additional options
pip install -e '.[opencompass]'   # Install OpenCompass backend
pip install -e '.[vlmeval]'       # Install VLMEvalKit backend
pip install -e '.[all]'           # Install all backends (Native, OpenCompass, VLMEvalKit)

๐Ÿš€ Quick Start

1. Simple Evaluation

To evaluate a model using default settings on specified datasets, follow the process below:

Install using pip

You can execute this command from any directory:

python -m evalscope.run \
 --model qwen/Qwen2-0.5B-Instruct \
 --template-type qwen \
 --datasets arc

Install from source

Execute this command in the evalscope directory:

python evalscope/run.py \
 --model qwen/Qwen2-0.5B-Instruct \
 --template-type qwen \
 --datasets arc

If prompted with Do you wish to run the custom code? [y/N], please type y.

Basic Parameter Descriptions

2. Parameterized Evaluation

If you wish to conduct a more customized evaluation, such as modifying model parameters or dataset parameters, you can use the following commands:

Example 1:

python evalscope/run.py \
 --model qwen/Qwen2-0.5B-Instruct \
 --template-type qwen \
 --model-args revision=master,precision=torch.float16,device_map=auto \
 --datasets gsm8k ceval \
 --use-cache true \
 --limit 10

Example 2:

python evalscope/run.py \
 --model qwen/Qwen2-0.5B-Instruct \
 --template-type qwen \
 --generation-config do_sample=false,temperature=0.0 \
 --datasets ceval \
 --dataset-args '{"ceval": {"few_shot_num": 0, "few_shot_random": false}}' \
 --limit 10

Parameter Descriptions

In addition to the three basic parameters, the other parameters are as follows:

3. Use the run_task Function to Submit an Evaluation Task

Using the run_task function to submit an evaluation task requires the same parameters as the command line. You need to pass a dictionary as the parameter, which includes the following fields:

1. Configuration Task Dictionary Parameters

import torch
from evalscope.constants import DEFAULT_ROOT_CACHE_DIR

# Example
your_task_cfg = {
        'model_args': {'revision': None, 'precision': torch.float16, 'device_map': 'auto'},
        'generation_config': {'do_sample': False, 'repetition_penalty': 1.0, 'max_new_tokens': 512},
        'dataset_args': {},
        'dry_run': False,
        'model': 'qwen/Qwen2-0.5B-Instruct',
        'template_type': 'qwen',
        'datasets': ['arc', 'hellaswag'],
        'work_dir': DEFAULT_ROOT_CACHE_DIR,
        'outputs': DEFAULT_ROOT_CACHE_DIR,
        'mem_cache': False,
        'dataset_hub': 'ModelScope',
        'dataset_dir': DEFAULT_ROOT_CACHE_DIR,
        'limit': 10,
        'debug': False
    }

Here, DEFAULT_ROOT_CACHE_DIR is set to '~/.cache/evalscope'.

2. Execute Task with run_task

from evalscope.run import run_task
run_task(task_cfg=your_task_cfg)

Evaluation Backend

EvalScope supports using third-party evaluation frameworks to initiate evaluation tasks, which we call Evaluation Backend. Currently supported Evaluation Backend includes:

Custom Dataset Evaluation

EvalScope supports custom dataset evaluation. For detailed information, please refer to the Custom Dataset Evaluation ๐Ÿ“–User Guide

Offline Evaluation

You can use local dataset to evaluate the model without internet connection.

Refer to: Offline Evaluation ๐Ÿ“– User Guide

Arena Mode

The Arena mode allows multiple candidate models to be evaluated through pairwise battles, and can choose to use the AI Enhanced Auto-Reviewer (AAR) automatic evaluation process or manual evaluation to obtain the evaluation report.

Refer to: Arena Mode ๐Ÿ“– User Guide

Model Serving Performance Evaluation

A stress testing tool that focuses on large language models and can be customized to support various data set formats and different API protocol formats.

Refer to : Model Serving Performance Evaluation ๐Ÿ“– User Guide

TO-DO List

Star History

Star History Chart