Awesome
<div align="center"> <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/Evalverse_White.png" width=300> <source media="(prefers-color-scheme: light)" srcset="assets/Evalverse_Color.png" width=300> <img alt="Evalverse" src="assets/Evalverse_Color.png" width=300> </picture>The Universe of Evaluation. All about the evaluation for LLMs. </br> Upstage Solar is powered by Evalverse! Try at Upstage Console!
π€HugginFace Space β’ πDocs β’ πPaper
Examples β’ FAQ β’ Contribution Guide β’ Contact β’ Discord
</div>π Newly updated
- [2024.05.10] LLM-Evaluation Report of Evalverse is now available on HuggingFace Space.
π Welcome to Evalverse!
Evalverse is a freely accessible, open-source project designed to support your LLM (Large Language Model) evaluation needs. We provide a simple, standardized, and user-friendly solution for the processing and management of LLM evaluations, catering to the needs of AI research engineers and scientists. We also support no-code evaluation processes for people who may have less experience working with LLMs. Moreover, you will receive a well-organized report with figures summarizing the evaluation results.
With Evalverse, you are empowered to
- access various evaluation methods without juggling multiple libraries.
- receive insightful report about the evaluation results that helps you to compare the varied scores across different models.
- initiate evaluation and generate reports without any code via Slack bot.
Architecture of Evalverse
<div align="center"><img alt="architecture" src="assets/architecture.png" width=700></div>Key Features of Evalverse
- Unified evaluation with Submodules: Evalverse extends its evaluation capabilities through Git submodules, effortlessly incorporating frameworks like lm-evaluation-harness and FastChat. Swiftly add new tools and keep pace with the latest in LLM evaluation.
- No-code evaluation request: With Evalverse, request LLM evaluations without any code, simply by sending
Request!
in a direct message or Slack channel with an activate Evalverse Slack bot. Enter the model name in the Huggingface hub or local model directory path in Slack, and let the bot handle the rest. - LLM evaluation report: Obtain comprehensive, no-code reports from Evalverse. Request with a simple command -
Report!
-, select the model and evaluation criteria, and receive detailed reports with scores, rankings, and visuals, all generated from the stored score database.
If you want to know more about Evalverse, please checkout our docs. </br>
By clicking below image, it'll take you to a short intro video!
</br>
π Installation
π Option 1: Git clone
Before cloning, please make sure you've registered proper SSH keys linked to your GitHub account.
1. Clone the Evalverse repository
- Notes: add
--recursive
option to also clone submodules
git clone --recursive https://github.com/UpstageAI/evalverse.git
2. Install requirement packages
cd evalverse
pip install -e .
π Option 2: Install via Pypi (WIP)
</br>Currently, installation via Pypi is not supported. Please install Evalverse with option 1.
π Configuration
You have to set an API key and/or Token in the .env
file (rename .env_sample
to .env
) to use all features of Evalverse.
- OpenAI API Key (required for
mt_bench
) - Slack BOT/APP Token (required for slack reporter)
OPENAI_API_KEY=sk-...
SLACK_BOT_TOKEN=xoxb-...
SLACK_APP_TOKEN=xapp-...
</br>
π Quickstart
More detailed tutorials are here.
- basic_usage.ipynb: Very basic usage, like how to use
Evaluator
for evaluation andReporter
for generating report. - advanced_usage.ipynb: Introduces methods for evaluating each benchmark and all benchmarks collectively.
π Evaluation
π« Evaluation with Library
The following code is a simple example to evaluate the SOLAR-10.7B-Instruct-v1.0 model on the h6_en
(Open LLM Leaderboard) benchmark.
import evalverse as ev
evaluator = ev.Evaluator()
model = "upstage/SOLAR-10.7B-Instruct-v1.0"
benchmark = "h6_en"
evaluator.run(model=model, benchmark=benchmark)
π« Evaluation with CLI
Here is a CLI script that produces the same result as the above code:
cd evalverse
python3 evaluator.py \
--h6_en \
--ckpt_path upstage/SOLAR-10.7B-Instruct-v1.0
π Report
Currently, generating a report is only available through the library. We will work on a Command Line Interface (CLI) version as soon as possible.
import evalverse as ev
db_path = "./db"
output_path = "./results"
reporter = ev.Reporter(db_path=db_path, output_path=output_path)
reporter.update_db(save=True)
model_list = ["SOLAR-10.7B-Instruct-v1.0", "Llama-2-7b-chat-hf"]
benchmark_list = ["h6_en"]
reporter.run(model_list=model_list, benchmark_list=benchmark_list)
<img alt="architecture" src="assets/sample_report.png" width=700>
Model | Ranking | total_avg | H6-ARC | H6-Hellaswag | H6-MMLU | H6-TruthfulQA | H6-Winogrande | H6-GSM8k |
---|---|---|---|---|---|---|---|---|
SOLAR-10.7B-Instruct-v1.0 | 1 | 74.62 | 71.33 | 88.19 | 65.52 | 71.72 | 83.19 | 67.78 |
Llama-2-7b-chat-hf | 2 | 53.51 | 53.16 | 78.59 | 47.38 | 45.31 | 72.69 | 23.96 |
π Supported Evaluations
We currently support four evaluation methods. If you have suggestions for new methods, we welcome your input!
Evaluation | Original Repository |
---|---|
H6 (Open LLM Leaderboard) | EleutherAI/lm-evaluation-harness |
MT-bench | lm-sys/FastChat |
IFEval | google-research/instruction_following_eval |
EQ-Bench | EQ-bench/EQ-Bench |
π Evalverse use-case
If you have any use-cases of your own, please feel free to let us know. </br>We would love to hear about them and possibly feature your case.
β¨ Upstage
is using Evalverse for evaluating Solar. </br>
β¨ Upstage
is using Evalverse for evaluating models at Open Ko-LLM Leaderboard.
π Contributors
<a href="https://github.com/UpstageAI/evalverse/graphs/contributors"> <img src="https://contrib.rocks/image?repo=UpstageAI/evalverse"/> </a>π Acknowledgements
Evalverse is an open-source project orchestrated by the Data-Centric LLM Team at Upstage
, designed as an ecosystem for LLM evaluation. Launched in April 2024, this initiative stands at the forefront of advancing evaluation handling in the realm of large language models (LLMs).
π License
Evalverse is completely freely-accessible open-source and licensed under the Apache License 2.0.
π Citation
If you want to cite our π Evalverse project, feel free to use the following bibtex. You can check our paper via link.
@misc{kim2024evalverse,
title={Evalverse: Unified and Accessible Library for Large Language Model Evaluation},
author={Jihoo Kim and Wonho Song and Dahyun Kim and Yunsu Kim and Yungi Kim and Chanjun Park},
year={2024},
eprint={2404.00943},
archivePrefix={arXiv},
primaryClass={cs.CL}
}